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THE  PROCESS  OF  CURRICULUM  INNOVATIONS  IN  THE  ARMY 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  research  was  in  response  to  a  request  from  the  United  States  Army  Training  and 
Doctrine  Command  (TRADOC)  to  examine  the  implementation  of  ALM  concepts  within 
institutional  courses  and  to  provide  guidance  to  TRADOC  on  how  to  facilitate  such  curriculum 
changes.  ALM  is  a  “paradigm  shift”  from  institutional  training  approaches  which  focus  heavily 
on  instructor-led  lectures.  The  Army  Learning  Concept  (ALC;  TRADOC  Pamphlet  525-8-2) 
contains  information  about  the  way  training  was  done  prior  to  ALM  and  why  ALM  is  needed. 
The  implementation  of  ALM  across  the  entire  Army  school  system  was  a  multi-year  effort  [e.g., 
in  depth  discussions  with  and  among  the  Centers  of  Excellence  (COEs),  dissemination  of 
guidance  documents,  further  articulation  and  operationalization  of  ALM  principles,  site  visits, 
special  conferences,  policy  revisions,  development  of  an  Army  Concept,  mobile  training  teams 
to  work  with  the  schools]  and  a  top  priority  for  TRADOC  during  that  timeframe  (2012-2015). 
Moreover,  the  TRADOC  G-3/5/7  understood  that  examining  the  extent  of  the  implementation 
was  a  necessary  first  step  in  determining  whether  the  ALC  objectives  were  being  met.  To 
examine  how  the  Centers  and  Schools  were  implementing  the  ALM,  the  U.S.  Army  Research 
Institute  (  ARI)  proposed  a  research  effort  centered  on  adaptation  of  innovation  concepts  per 
ALM  implementation  with  respect  to  the  courses  in  the  “Shot-in-the-Arm  (STTA)”  project  across 
several  Centers  and  Schools.  The  purpose  was  to  determine  which  factors  facilitated  success 
during  ALM  SITA  course  revisions,  whether  the  implementation  process  was  working,  the 
effects  on  personnel  and  resources,  and  the  stage  of  innovation  adaptation  for  each  course. 

Procedure: 

Ten  courses  were  examined;  they  were  selected  by  TRADOC  from  the  courses  which  the 
CoEs  nominated  as  part  of  TRADOC’ s  SITA  program.  The  courses  existed  at  seven  CoEs  and 
Schools.  They  included  two  Advanced  Individual  Training  courses,  seven  Noncommissioned 
Officer  Education  System  (NCOES)  courses  (both  Advanced  Leader  Courses  and  Senior  Leader 
Courses),  and  one  Officer  Basic  Course.  A  total  of  72  individuals  participated  in  the 
interviews/focus  groups.  The  specific  duty  positions  included  middle  managers,  course 
managers,  training  developers,  instructional  designers,  senior  instructors,  instructors,  chief  of 
training,  unit  leaders,  quality  assurance  personnel,  and  Staff  and  Faculty  personnel.  In  addition, 
graduates  of  one  course  were  followed  to  their  next  duty  position.  Site  visits  started  in  August 
2014  and  were  completed  in  March  2015. 

Findings: 

The  courses  were  divided  into  three  stages  of  innovation:  a  Sustain  phase  where  the 
courses  had  been  implemented  at  least  three  times  and  a  steady  state  achieved;  an  Execute  phase 
where  courses  had  been  implemented  at  least  once  but  steady  state  had  not  been  achieved;  and  a 
Plan/Develop  stage  where  the  course  had  not  yet  been  executed.  Based  on  the  Sustain  courses 
timelines,  the  time  required  to  make  major  changes  in  a  course  and  to  ensure  the  changes  are 
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successful  was  estimated  to  be  1.5  to  2  years.  Major  facilitating  factors  were  that  participants 
perceived  an  advantage  from  implementing  the  ALM,  pilots  of  the  course  or  parts  of  a  course 
were  executed  (as  opposed  to  revising  the  entire  course  at  one  time)  which  created  a  positive 
feedback  loop  between  training  developers  and  instructors,  and  there  was  compatibility  with 
some  instructional  techniques  already  being  implemented. 

Factors  that  inhibited  timely  development  and  implementation  related  to  limited  ALM 
training  of  personnel  in  some  courses,  requirements  to  make  other  course  changes  which  were  in 
addition  to  ALM,  and  difficulty  in  modifying  lesson  plans  and  course  implementation  to  adapt 
effectively  to  individual  differences  in  students’  prior  knowledge  and  background.  That  is, 
tailored  training  was  of  concern  when  the  students  differed  substantially  in  prior  experiences  and 
relevant  knowledge.  Across  all  courses,  training  developers  and  instructors  had  greater 
workloads,  and  more  preparation  was  required  by  instructors. 

In  general,  participants  reacted  favorably  to  the  ALM  concepts,  stressing  increased 
student  motivation,  involvement  and  class  participation.  In  terms  of  training  techniques,  the  use 
of  PowerPoint  (a  “sage  on  a  stage”)  presentations  was  replaced  by  methods  to  facilitate  learning 
and  thinking  via  the  instructor  and/or  through  small  groups,  peer-to-peer  interactions,  and  hands- 
on  practical  exercises.  More  responsibility  was  placed  on  the  students  with  home-work  and 
read-ahead  materials  as  well  as  a  change  in  test  procedures.  PowerPoint  slides  were  greatly 
reduced  but  not  necessarily  eliminated.  The  training  developers  and  instructors  worked  very 
hard  at  implementing  ALM  concepts  and  contributed  to  the  successes  that  occurred. 

A  key  factor  underlying  the  majority  of  the  research  findings  and  in  sustaining 
curriculum  changes  is  how  well  instructors  are  trained  to  implement  the  types  of  techniques 
required  by  the  ALM.  As  new  instructors  are  assigned  to  courses,  these  instructors  need  to  know 
the  overall  purpose  of  the  ALM  and  the  specific  techniques,  methods,  and  processes  for 
implementing,  sustaining,  and  furthering  the  ALM  tenets  within  Army  courses. 

A  Roadmap  for  Success  stresses  the  importance  of  planning  to  include  planning  for 
performance  assessment  and  involving  all  key  players,  monitoring  course  execution  to  include 
assigning  an  observer  to  the  course  who  can  provide  instructor  feedback  directly  to  the  training 
developers,  and  providing  resources  for  instructor  training  which  is  critical  for  sustainment  of  the 
course  revisions  and  further  implementation  of  ALM  concepts. 

Utilization  and  Dissemination  of  Findings: 

The  results  of  this  research  were  briefed  to  the  Army  Learning  Coordination  Council 
(ALCC)  Working  Group  and  the  (ALCC)  Principals  in  September  2015.  Prior  to  these  briefs,  the 
results  were  provided  to  all  of  the  Centers  and  Schools  who  participated  in  the  data  collection. 
Several  of  the  Centers  and  Schools  provided  valuable  feedback  which  was  incorporated  into  the 
ALCC  briefs.  The  findings  of  the  present  research  are  being  utilized  in  a  follow-on  ARI  project 
which  is  developing  a  tool  to  support  curriculum  developers  and  instructors  in  selecting  the  most 
effective  instructional  methods  for  their  classes. 
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The  Process  of  Curriculum  Innovations  in  the  Army 


The  Army  Learning  Model 

In  2011  the  Department  of  the  Army’s  Training  and  Doctrine  Command  (TRADOC) 
published  TRADOC  Pamphlet  525-8-2,  The  U.S.  Army  Learning  Concept  for  2015.  The  concept 
describes  the  Army  Learning  Model  (ALM)  that  meets  the  Army’s  need  to  develop  adaptive, 
critical  thinking  Soldiers  and  Leaders  capable  of  meeting  the  challenges  of  operational 
adaptability  in  an  era  of  persistent  conflict.  It  is  a  learning  model  that  adapts  to  fluctuations  in 
learning  time  and  maximizes  opportunities  to  master  fundamental  competencies.  It  is  open  to 
inventiveness,  to  input  of  learner  knowledge,  and  advances  in  learning  technologies  and 
methods.  It  describes  a  learning  continuum  that  meshes  self-development,  institutional 
instruction,  and  operational  experience.  This  is  a  learner-centric  continuum  that  begins  when  an 
individual  joins  the  Army  and  ends  at  retirement.  The  learning  model  enhances  the  rigor  and 
relevance  of  individual  learning  through  assessment  of  21st  Century  Soldier  Competencies  that 
enable  success  across  full-spectrum  operations.  These  nine  Soldier  Competencies  (e.g., 
character  and  accountability,  tactical  and  technical  competence)  are  to  be  instilled  and  reinforced 
in  Soldiers  during  their  careers  (see  Appendix  A  for  competency  descriptions). 

The  ALM  is  the  operational  term  for  the  continuous  adaptive  learning  model  just 
described.  TRADOC  Schools  are  incrementally  implementing  the  ALM  with  changes  to 
instructional  strategies  that  shift  from  slide-based  lectures  to  facilitated,  collaborative  learning 
events  and  that  engage  learners  by  employing  digital  learning  content  and  relevant  operational 
scenarios  and  blended  learning  approaches.  Such  shifts  in  instructional  strategies  mean  a  major 
paradigm  shift  in  many  courses.  In  FY 13  and  FY 14,  the  Commanding  General  of  TRADOC 
reinforced  ALM  2015  implementation  efforts  by  requiring  each  Center  of  Excellence  (CoE)  to 
revise  at  least  one  course  program  of  instruction  (POI)  and  associated  lesson  plans  subsequent  to 
its  key  personnel  (e.g.,  course  managers,  training  developers,  instructional  designers,  instructors, 
quality  assurance  personnel)  attending  mobile  training  teams  (MTTs)  on  ALM  principles.  This 
initiative  was  named  the  “Shot-In-The-Arm  (SITA)”  project,  and  each  revised  course  was 
required  to  model  ALM  guiding  concepts  and  focus  on  the  development  of  the  21st  Century 
Soldier  Competencies.  Although  some  of  the  CoE  course  revisions  may  have  reflected 
technology-delivered  instruction,  the  purpose  of  the  research  project  was  to  identify  factors 
which  facilitated  success  and  provide  lessons  learned  and  recommendations  regarding  the 
broader  implementation  of  ALM. 


Research  Purpose 

The  research  was  in  response  to  a  request  from  TRADOC  to  examine  the  implementation 
of  ALM  concepts  within  institutional  courses  and  to  provide  guidance  to  TRADOC  on  how  to 
facilitate  such  curriculum  changes.  ALM  was  a  “paradigm  shift”  from  the  current  institutional 
training  approaches  which  focused  heavily  on  instructor- led  lectures.  The  Army  Learning 
Concept  (TRADOC  Pamphlet  525-8-2)  contains  information  about  the  way  training  was  done 
prior  to  ALM  and  why  ALM  was  needed.  The  implementation  of  ALM  across  the  entire  Army 
school  system  was  a  multi-year  effort  (e.g.,  in  depth  discussions  with  and  among  the  COEs, 
dissemination  of  guidance  documents,  further  articulation  and  operationalization  of  ALM 
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principles,  site  visits,  special  conferences,  policy  revisions,  development  of  an  Army  Concept, 
mobile  training  teams  to  work  with  the  schools)  and  top  priority  for  TRADOC  during  that 
timeframe.  Moreover,  the  TRADOC  G- 3/5/7  understood  that  examining  the  extent  of  the 
implementation  was  a  necessary  first  step  in  determining  whether  the  Army  Learning  Concept 
objectives  were  being  met.  To  examine  how  the  Centers  and  Schools  were  implementing  the 
ALM,  the  U.S.  Army  Research  Institute  proposed  a  research  effort  centered  on  adaptation  of 
innovation  concepts  per  ALM  implementation  with  respect  to  the  courses  in  the  SITA  project 
across  several  Centers  and  Schools.  The  purpose  was  to  determine  which  factors  facilitated 
success  during  ALM  SITA  course  revisions,  whether  the  implementation  process  was  working, 
the  effects  on  personnel  and  resources,  and  the  stage  of  innovation  adaptation  for  each  course. 

The  United  States  Army  has  a  formal  institutional  education  and  training  function 
(reference  The  Army  School  System,  Department  of  the  Army,  2010,  TRADOC  Regulation  350- 
18).  Education  and  training  are  provided  at  many  locations  and  address  the  requirements  of 
Soldiers,  noncommissioned  officers  (NCOs),  and  officers  during  their  Army  careers.  Instructors 
can  be  military,  civilian,  or  both.  The  courses  vary  in  content,  graduation  requirements,  student 
population,  and  length.  A  specific  course  is  a  reflection  of  a  unique  intersection  of  required 
tasks,  skills,  knowledge,  and  competencies  in  a  given  domain  dependent  on  the  individual’s 
career  progression.  Consequently,  when  a  substantial  change  to  instructional  approaches  is 
proposed,  the  process  of  making  those  changes  can  vary  greatly.  The  courses  participating  in  the 
research  were  selected  from  the  pool  of  courses  nominated  for  SITA  by  the  CoEs,  and  final 
selections  were  based  on  recommendations  from  TRADOC.  Although  the  research  was 
executed  in  the  context  of  the  ALM,  the  findings  have  general  application  to  other  large 
curriculum  changes  in  the  Army.  The  conceptual  background  and  lessons  learned  presented  here 
provide  a  picture  of  the  dynamics  involved  with  making  major  curriculum  changes  and  the 
factors  that  can  facilitate  and  inhibit  that  process. 

Conceptual  Background  for  the  Research 

Adoption  of  ALM  concepts  implies  a  major  behavioral  change  for  most  instructors  and 
also  on  how  courses  are  developed.  Adoption  does  not  mean  simply  replacing  a  single  lesson 
plan  which  is  typically  presented  in  a  face-to-face  situation  with  an  on-line  learning  session  or  a 
computer-based  session.  Rather,  for  many  courses,  the  ALM  approach  applies  to  the  entire 
course,  shifting  from  lecture-based  presentations  to  facilitated  discussions,  and  requires  a 
different  set  of  instructor  skills,  a  different  type  of  instructor  preparation  /  materials  /  testing,  etc. 
It  often  places  more  emphasis  upon  the  students  for  being  responsible  for  their  own  learning. 
Given  this  context,  the  ALM  changes  in  general,  and  the  specific  changes  in  the  SITA- 
designated  courses  were  viewed  as  types  of  “innovations.”  Consequently,  the  primary 
framework  for  examining  the  courses  was  based  on  the  “diffusion  of  innovations”  literature 
(Rogers,  2003),  specifically  the  research  related  to  adopting  innovations  within  organizations, 
summarized  in  Rogers’  book  Diffusion  of  Innovations. 

Although  Rogers  focused  primarily  on  innovations  in  society  as  a  whole  (e.g.,  adoption 
of  hybrid  seed  corn  in  the  1930s-1950s,  Toshiba’s  creation  of  the  laptop  computer  in  the  1980s, 
adoption  of  the  internet  from  1970  to  2002,  and  the  failure  to  adopt  the  Dvorak  keyboard  in  the 
1930s),  he  also  examined  the  innovation  process  within  organizations.  These  concepts  have 
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direct  application  to  the  Army’s  institutional  training  framework.  Within  an  organization,  the 
innovation  decision  is  often  made  by  a  person  in  authority  (e.g.,  adoption  of  communication 
technologies  by  banks  or  of  medical  technologies  by  hospitals)  as  opposed  to  being  an  optional 
decision  to  try  an  innovation  by  an  individual  (e.g.,  deciding  to  buy  a  cellular  telephone).  Rogers 
called  such  decisions  “authority  innovation-decisions.”  With  such  decisions,  employees  of  an 
organization  must  comply.  Rogers  also  indicated  that  once  a  decision  is  made  to  adopt, 
implementation  may  not  automatically  follow,  and  the  innovation  process  within  organizations 
can  be  complex.  Furthermore,  the  organizational  structures  can  vary,  resulting  in  differences  in 
the  authority  structure  and  formal  policies/procedures.  Rogers  noted  that  informal  processes  and 
practices  within  organizations  also  influence  the  adoption  process.  For  the  present  research,  the 
expectation  was  that  the  innovation  process  would  likely  differ  among  the  Army  Schools  /  CoEs, 
given  the  different  organizational  structures  across  the  CoEs,  differences  in  internal  processes 
and  practices,  and  differences  in  course  objectives  and  student  populations. 

Stages  of  Innovation 

Rogers’  five  stages  of  innovation  are  described  briefly  in  Table  1  below.  The  cross-walk 
between  these  five  stages  and  the  integration  of  the  ALM  into  Army  courses  is  also  shown. 

Table  1 

Rogers’  Stages  of  Innovation  and  Their  Application  to  Innovations  in  Army  Courses 


Rogers’  Stages  of  Innovation 

Relationship  to  ALM  Integration 

Agenda  Setting: 

Problem  defined  which  identifies  a  need  to 
change,  and  search  initiated  to  identify 
innovations  that  would  be  useful  solutions  to  the 
problem 

Decision-making  -  Course  Selection: 

Who,  when  and  why  the  course  was  selected  for 
TRADOC’s  SITA  effort;  identification  of  training 
gaps;  lines  of  communication  between  subordinate 
units  at  this  stage 

Matching: 

Match  the  problem  to  appropriate  solution, 
examine  feasibility  of  solution,  and  anticipate 
benefits  and  problems  to  be  encountered  when 
implemented 

Planning  and  Identifying  Solutions: 

Determining  how  to  best  implement  ALM 
concepts  in  the  course;  how  to  best  address 
training  gaps;  proposed  solutions;  complexity  of 
the  problem  to  be  addressed. 

Redefining/structuring: 

Redefine  innovation  to  meet  organization’ s  needs 
and  structure 

Developing  Course  Revisions: 

Actual  development  of  course  revisions; 
complexity  of  revisions;  interactions  between 
training  developers  and  instructors;  resources 
needed  and  available.  Re-invention  may  occur 
here. 
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Clarifying: 

Innovation  now  put  into  widespread  use,  new  idea 
clearer  to  users,  misunderstandings  clarified,  and 
common  understandings  achieved 


Routinizing: 

Innovation  now  incorporated  into  regular 
activities  of  the  organization.  Critical  issue  is 
whether  it  will  be  sustained. 


Implementing/Executing  the  Revised  Course: 

Instructors  execute  the  revised  course;  how 
revisions  affect  the  instructors;  coordination 
between  instructors  and  training  developers  to 
revise  course  as  necessary;  execution  success  and 
issues. 

Sustaining  the  Revised  Course: 

Course  revisions  stabilized,  reached  a  steady  state; 
decision-makers,  training  developers  and 
instructors  satisfied  with  changes  and  adequacy  of 
implementation.  Sustainment  plans  if  sustainment 
not  yet  achieved. 


Rate  of  Adoption  of  Innovations 

The  research  on  diffusion  of  innovations  also  identifies  factors  that  facilitate  and  inhibit 
changes  in  organizations.  Rogers  (2003)  cited  five  major  characteristics  of  an  innovation  that 
impact  the  rate  of  its  adoption.  These  are  cited  in  Table  2  but  not  in  order  of  their  importance  or 
relative  impact.  All  factors,  except  complexity,  relate  positively  to  the  rate  of  adoption. 

Table  2 

Rogers  ’  Five  Major  Characteristics  that  Impact  the  Rate  of  Adoption  of  Innovations 


Rate  of  Adoption  Factor 

Relative  Advantage: 

The  degree  to  which  an  innovation  is  perceived  as 
better  than  the  idea  it  supersedes.  The  important 
point  is  that  the  innovation  is  perceived  as  better, 
not  that  it  has  necessarily  been  shown  to  be  better. 
“Better”  may  mean  savings  of  time  and  effort, 
economic  profits,  low  cost,  etc. 


Relationship  to  ALM  Integration 

Relative  advantage  could  be  reflected  in  the  extent 
to  which  course  managers,  training  developers 
and/or  instructors  perceived  an  advantage  in 
modifying  the  mode  of  instruction,  e.g.,  student 
skills  /  knowledge  gained,  ability  of  students  to 
think,  understand  and  retain  subject  matter; 
advantages  of  greater  student  responsibility. 


Compatibility: 

The  degree  to  which  the  innovation  is  similar  to, 
overlaps  with,  or  is  consistent  with  adopters’ 
existing  values,  past  experiences,  needs,  and/or 
existing  norms. 


Compatibility  could  be  reflected  in  the  extent  to 
which  training  developers  and  instructors 
perceived  ALM  techniques/approaches  as 
consistent  with  changes  they  thought  were  needed 
(e.g.,  to  address  a  training  gap),  were  consistent 
with  changes  they  were  considering,  or  consistent 
with  similar  changes  made  in  other  courses. 
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Trialability: 

The  extent  to  which  an  innovation  can  be 
experimented  with  on  a  limited  basis.  Can  it  be 
tried  out?  Trialability  exposes  users  to  the 
innovation  on  a  gradual  basis  and  therefore 
presents  less  uncertainty  to  individuals  who  must 
use  the  innovation. 


Revised  courses  or  parts  of  the  courses  could  be 
tried  out  relatively  early  in  the  development 
process  to  determine  whether  the  revisions  were 
satisfactory.  In  other  words,  pilots  could  be 
conducted. 


Observability: 

The  extent  to  which  the  results  of  implementing 
the  innovation  are  visible  to  others,  the  more 
likely  individuals  are  to  adopt.  Observability 
removes  doubts  regarding  whether  the  innovation 
works. 

Complexity: 

The  difficulty  individuals  have  with 
understanding  and  using  the  innovation.  The 
more  complex  the  innovation,  then  the  slower  the 
rate  of  adoption. _ 


Courses  could  be  observed  or  monitored  (e.g., 
instructor  implementation  of  ALM  and  student 
behaviors/reactions).  The  question  was  whether 
this  actually  would  occur  in  the  curriculum 
development  process. 

Complexity  could  be  reflected  in  the  difficulty 
instructors  had  in  switching  from  lecturing  to 
facilitating  discussions.  Training  developers 
could  have  difficulty  incorporating  good  problem¬ 
solving  or  thinking  exercises. 


Rogers  cited  three  other  factors  which  were  perceived  as  directly  relevant  to  the  ALM 
effort.  These  are  summarized  in  Table  3. 

Table  3 

Additional  Factors  that  Can  Impact  the  Rate  of  Adoption  of  Innovations 


Additional  Factors 


Relationship  to  ALM  Integration 


Reinvention: 

The  degree  to  which  an  innovation  is  changed  or 
modified  by  a  user  in  the  process  of  adoption  and 
implementation.  Many  innovations  are  re-invented  as 
adopters  implement  them  in  a  variety  of  ways,  thus 
changing  the  innovation  during  the  diffusion  process. 
An  innovation  diffuses  more  rapidly  when  it  can  be 
re-invented  and  that  adoption  is  more  likely  to  be 
sustained. 

Coordination  between  units: 

The  degree  of  interconnectedness,  coordination,  or 
interactions  between  individuals  and  /or  units  within 
an  organization  influences  the  rate  of  adoption.  New 
ideas  can  be  disseminated  more  quickly  and  easily  to 
the  degree  that  interconnectedness  exists.  Also  when 
an  organization  can  apply  additional  resources  to  an 
innovation,  then  adoption  occurs  more  readily. 


Given  that  ALM  concepts  are  relatively  broad 
and  not  entirely  prescriptive,  ALM  could  be 
defined  and  applied  in  different  ways  by  the 
Centers  and  Schools,  depending  on  their 
particular  circumstances.  Schools  could  also 
feel  enabled  to  change  or  re-invent  with  time 
and  experience  with  the  course. 


As  the  CoEs  have  different  organizational 
structures,  both  the  formal  and  informal  paths 
of  communication  among  the  key  training 
personnel  could  differ  and  could  have 
different  impacts  upon  the  innovation  process. 
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Champion: 

A  champion  is  an  individual  who  fully  supports  the 
innovation  and  Pies  to  overcome  resistance  to  it 
within  the  institution.  Having  a  champion  facilitates 
the  success  of  an  innovation.  Champions  do  not  need 
to  be  powerful  individuals.  They  are  often  middle 
managers;  individuals  who  are  adept  at  working  with 
others  and  skilled  in  negotiating  and  persuading.  A 
champion  focuses  on  the  innovation  at  the  start  and 
continues  to  be  involved  to  ensure  successful 
implementation. 


Champions  of  the  ALM  implementation  could 
be  any  of  the  individuals  involved  in  the  SIT  A 
effort  such  as  unit  leaders,  course  managers, 
training  developers,  and/or  instructors. 


Research  Objectives 

The  first  objective  was  to  determine  the  stage  of  innovation  for  each  course,  determine 
which  factors  facilitated  success  with  the  ALM  course  modifications,  determine  which  factors 
restrained  success,  and  provide  lessons  learned  /  recommendations  regarding  changes  in  other 
courses.  The  factors  which  facilitated  and  inhibited  success  were  based  on  Rogers’  model  (see 
Tables  1,  2,  and  3). 

The  second  objective  was  to  develop  a  framework  for  gathering  evidence  on  ALM 
implementation  to  enable  decision-makers  to  determine  whether  an  implementation  process 
worked,  the  impact  of  the  change  process  on  personnel  involved  with  the  implementation  (e.g., 
instructors,  training  developers,  students),  and  major  resources  required.  Guidelines  to  assist 
decision-makers  in  assessing  the  progress,  in  facilitating  changes,  and  in  documenting  benefits  of 
the  changes  also  were  to  be  included  in  the  framework. 

The  third  objective  was  to  document  the  substantive  changes  in  the  courses  from  the 
perspective  of  ALM  concepts  and  principles.  This  objective  included  determining  changes  in 
instructional  delivery  methods  (e.g.,  technology)  and/or  in  major  training  approaches,  and 
identifying  which  21st  Century  Soldier  Competencies  were  reflected  in  the  approaches  taken. 

Lastly,  a  unique  feature  of  the  research  was  that  the  graduates  from  one  of  the  courses  in 
the  effort  were  followed  to  their  next  duty  station  to  gain  insights  regarding  the  impact  of  the 
revised  course  on  their  performance  in  the  field.  Comparisons  were  made  with  individuals  who 
had  graduated  prior  to  the  revised  course.  In  this  report,  the  general  lessons  regarding 
implementation  of  ALM  concepts/principles  and  research  methodology  are  presented.  Results 
unique  to  the  course  are  not  cited. 


Method 

The  ten  courses  examined  for  the  present  research  were  selected  by  TRADOC  from  the 
courses  which  the  CoEs  nominated  as  part  of  TRADOC’s  SITA  program.  The  courses  existed  at 
seven  CoEs  and  Schools.  The  ten  courses  included  two  Advanced  Individual  Training  courses, 
seven  NCO  professional  development  courses  (both  Advanced  Leader  Courses  and  Senior 
Leader  Courses),  and  one  Officer  Basic  Course. 
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Participants 


A  total  of  72  individuals  participated  in  the  interviews/focus  groups.  The  specific  duty 
positions  included  middle  managers,  course  managers,  training  developers,  instructional 
designers,  senior  instructors,  instructors,  chief  of  training,  unit  leaders,  quality  assurance 
personnel,  and  Staff  and  Faculty  personnel.  Middle  managers  were  individuals  who  monitored 
several  course  managers;  this  position  did  not  exist  at  all  Schools. 

A  background  survey  (see  Appendix  B)  was  completed  by  63%  ( n  =  45)  of  the 
participants,  who  then  completed  a  questionnaire  appropriate  to  their  roles  and  responsibilities  in 
the  innovation  process  and/or  were  interviewed  about  their  roles  and  responsibilities.  The 
planning  questionnaire  was  completed  by  ten  individuals.  The  developing  revisions 
questionnaire  was  completed  by  23  individuals. 

Site  visits  started  in  August  2014  and  were  completed  in  March  2015.  There  was  one 
major  limitation  of  the  effort.  The  surveys  and  interviews  were  conducted  at  a  point  in  time 
when  some  key  individuals  in  the  planning  or  development  process  had  left  their  positions. 

Thus,  relevant  input  regarding  some  stages  (e.g.,  planning  and  development  start  dates)  had  to  be 
obtained  from  individuals  who  were  less  knowledgeable  because  they  were  not  in  the  current 
position  during  the  decision-making  processes  for  the  selection  of  a  course  for  the  SITA  effort 
and/or  not  involved  in  the  planning  processes  for  the  revisions.  The  extent  to  which  this 
occurred  varied  with  the  course. 

Questionnaires  and  Interviews 

Background  information.  The  background  questionnaire  (see  Appendix  B)  asked 
participants  to  indicate  their  duty  position  and  their  time  in  this  position,  and  to  indicate  the 
SITA  training  they  had  attended.  They  were  also  asked  about  the  extent  of  their  role  in  each  of 
the  five  stages  of  innovation,  the  timing  of  each  stage  (if  known),  and  to  select  the  best 
description  of  the  status  of  that  stage.  Figure  1  illustrates  the  type  of  scale  used  to  describe  the 
status  of  a  stage  (e.g.,  Stage  4).  Similar  three-category  rating  scales  were  used  to  determine  the 
degree  of  progress  for  each  of  the  other  stages.  Lastly  participants  indicated  their  interaction  and 
communication  with  individuals  in  other  key  duty  positions  during  the  course  revisions  and 
implementation  processes.  Interview  questions  which  supplemented  the  questionnaire  were  also 
developed  (see  Appendix  B).  The  implementation  timelines  were  established  from  pooling  the 
questionnaire  and  interview  data. 
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CHANGE  ACTIVITIES 


Circle  the  cells  that  best  describe  the  implementation/ 
execution  stage. 


Have  not  yet 
implemented 
the  revised 
course. 


Have  not  collected 
feedback  regarding 
">  the  course 
revisions 


Figure  1.  Illustration  of  rating  scales  in  the  background  questionnaire  (status  of  the  Execution 
stage  -  Stage  4) 


If  implemented,  modify 
course  from  feedback. 

Collect  obtain  feedback 
from  instructors  and/or 
students  and  further  revise 
the  course. 


Have  completed 
one  iteration  of 
the  revised 
course. 


Have  completed 
more  than  one 
iteration  of  the 
revised  course 


Have  collected 
course  feeback, 
but  have  not 
made  additional 
revisions  to  the 
course 


Have  collected 
course  feedback  and 
have  made 
additional  revisions 
to  the  course. 


Implement  change  or 
course  revisions. 

Execute  course  revisions 


Stages  of  innovation  questions.  It  was  acknowledged  that  there  could  be  overlap  among 
the  innovation  stages.  We  also  expected  that  individuals  interviewed  would  have  different  roles 
in  each  stage  and  would  not  necessarily  have  a  role  in  every  stage.  Even  though  each  individual 
was  asked  about  his/her  role,  more  extensive  surveys  and  interviews  were  conducted  with 
individuals  with  primary  roles  during  a  specific  stage.  If  an  individual  indicated  that  he  or  she 
did  not  have  a  role  in  a  specific  phase,  then  they  did  not  address  that  phase. 

Interview  protocols  were  developed  for  each  stage.  Questionnaires  were  developed  for 
only  Stages  2  and  3  (Planning  and  Developing).  All  instruments  are  in  Appendix  B.  An 
overview  of  the  questions  is  in  Table  4. 
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Table  4 

Primary  Questions  About  Each  Innovation  Stage 


Stage 

Primary  Questions  /  Topics 

Decision¬ 

making 

Who  made  the  decision  to  select  the  course  for  SITA 

How  many  courses  were  considered 

Primary  reason  for  selection 

Planning  & 
Identifying 
Solutions 

Plans  made  for  the  change 

Seven  dimensions  to  estimate  the  nature  of  change 

Degree  of  change 

Difficulty  of  making  the  changes 

Changes  in  instructor  behavior/procedures 

Time  requirements 

Manpower  requirements 

Workload 

Costs 

Questions  on  instructional  techniques  and  Soldier  competencies 

Developing 

Course 

Revisions 

Coordination  between  planners  and  training  developers 

Where  most  of  the  changes  were  made  (e.g.,  instructional  delivery,  problem¬ 
solving  exercises,  tailored  training) 

Actual  degree  of  change  on  the  same  seven  dimensions  asked  of  the 
planning  stage 

Questions  on  instructional  delivery,  manpower  and  time  requirements 

Implementing/ 
Executing  the 
Revised 

Course 

Instructor  experience  with  the  revised  course  and  the  legacy  course 

Interactions  with  training  developers 

Instructor  preparation  for  the  revised  course 

Changes  in  instructor  behavior  with  students,  changes  in  workload 

Maintenance  of  instructor  expertise 

Sustaining  the 
Revised 

Course 

Plans  for  preparation/training  of  instructors;  how  to  maintain  expertise 
Obtaining  student  reactions  to  course  and  indices  of  student  performance 
Lessons  learned  from  courses  that  had  been  executed  several  times 

As  indicated  in  Table  4,  seven  dimensions  were  used  in  both  the  planning  and  developing 
questions  to  determine  the  nature  of  the  changes  from  the  legacy  to  the  revised  course.  For  each 
dimension,  a  specific  scale  was  developed.  The  rating  scales  for  three  of  these  dimensions  are 
shown  in  Figure  2.  All  rating  scales  are  in  Appendix  B. 
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Minor  and  few  Somewhat  difficult 

changes  needed  in  for  the  instructor.  Substantial  changes 

r _ ,  instructional  Most  desired  in  instructional 

Change  in  techniques  from  the  changes  built  upon  techniques,  training 

instructor  prior  course;  no  new  instructional  required  to  prepare 

behavior/  instructor  training  techniques  instructors  for  most 

Drocedures  (e.g.,  removal  of  previously  used.  of  the  course  (e.g., 

L? - - -  PowerPoint  slides,  Instructor  training  hi§h  direct  instmction 

more  repetition  of  needed  for  some  to  primarily 

exercises  to  enhance  blocks  of  instruction  facilitator,  focus  on 

learning  and  (e.g.,  desk-top  technical  skills  on|Y t0 

retention)  simulations  or  also  focusing  on 

-  computer-based  decision-making) 

training  to  virtual 
simulations,  insertion 
of  more  complex 
problem-solving 
exercises) 

Figure  2.  Scales  used  to  rate  three  dimensions  on  the  nature  of  course  changes  (Planning  -  Stage 

2) 


Minor  and  few 
changes  needed  in 
instructional 
techniques  from  the 
prior  course;  no  new 
>  instructor  training 
(e.g.,  removal  of 
PowerPoint  slides, 
more  repetition  of 
exercises  to  enhance 
learning  and 
retention) 


Substantial  changes 
in  instructional 
techniques;  training 
required  to  prepare 
instructors  for  most 
of  the  course  (e.g., 
high  direct  instruction 
to  primarily 
facilitator,  focus  on 
technical  skills  only  to 
also  focusing  on 
decision-making) 
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The  interview  /  questionnaire  process.  During  the  interviews,  one  individual  asked 
questions  while  another  took  notes.  Both  individuals  then  reviewed  the  notes  for  completeness 
and  clarity.  The  interviews  were  used  to  verify  timeline  information  and  other  information 
obtained  from  the  background  questionnaire. 

A  codebook  (Appendix  C)  was  developed  for  coding  the  interview  records.  Two 
individuals  coded  each  interview.  In  many  instances,  a  given  comment  could  legitimately  be 
placed  in  more  than  one  category.  Any  direct  contradictions  in  the  codes  were  resolved  between 
the  two  coders.  The  major  categories  in  the  codebook  were: 

•  Shot-in  the- Arm  initiative 

•  Course  selection 

•  Role  of  key  personnel 

•  Instructors 

•  Knowledge  of  ALM 

•  Communication  between  stakeholders 

•  Rate  of  adoption  factors 

•  2 1 st  century  S  oldier  competencies 

•  Types  of  revisions 

•  Training  development  capability 

•  Outcomes 

For  each  major  category,  there  were  subcategories.  These  subcategories  were 
particularly  critical  in  coding  the  rate  of  adoption  factors.  For  example,  the  complexity  factor 
was  examined  in  terms  of  the  general  increase  in  workload  to  make  the  changes  (did  all  course 
lessons  have  to  be  changed  or  only  some  of  the  lessons),  degree  of  difficulty  in  making  exercises 
consistent  with  ALM  goals,  changes  in  the  types  of  tests  administered,  and  additional 
complexities  because  of  unexpected  changes  in  subject  matter  content. 

Follow-on  of  Course  Graduates 

Graduates  of  one  revised  course  were  followed  to  their  duty  station.  These  graduates 
were  interviewed,  as  well  as  their  leaders  and  graduates  who  had  received  the  “legacy”  course. 
Questions  focused  on  their  reactions  to  the  course  and  the  applicability  of  what  they  learned  to 
their  current  duty  positions  (e.g.,  what  were  the  most  valuable  parts  of  the  course,  where  could 
improvements  be  made,  time  required  to  assume  their  duties).  Similar  questions  were  asked  of 
the  graduates  of  the  “legacy”  course.  Leaders  were  asked  whether  they  perceived  differences  in 
the  skills,  expertise,  professionalism,  and  leadership  qualities  of  the  two  groups  of  graduates. 

The  interview  protocol  is  not  provided  in  this  report  to  prevent  disclosure  of  the  course  which 
was  investigated. 
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Results 


The  results  are  presented  by  the  stage  of  innovation  of  each  course  when  the 
questionnaires  and  interviews  were  conducted,  rather  than  by  the  total  sample.  The  courses  were 
divided  into  three  stages  based  on  Rogers’  framework: 

•  Sustain:  The  courses  had  been  executed  at  least  three  times  and  those  interviewed  felt 
a  steady  state  had  been  reached;  three  courses  were  in  this  category. 

•  Execute:  The  courses  had  been  executed  at  least  once,  but  changes  were  still  in  progress; 
four  courses  were  in  this  category. 

•  Plan/Dev:  The  courses  had  not  been  executed.  Typically  development  was  not 
complete  and/or  no  course  had  yet  been  piloted/implemented;  three  courses  were  in 
this  category. 

More  detail  on  these  course  stages  is  in  the  section  on  timelines. 

Participants  and  Courses 

The  numbers  of  participants  by  duty  position  and  innovation  stage  are  shown  in  Table  5. 


Table  5 

Number  of  Participants  Interviewed  by  Duty  Position  and  Innovation  Stage 


Duty  Position 

Innovation 

Stage 

Middle  /  Course  Manager  / 
Training  Developer  / 
Instructional  Designer 

Instructors  /  Sr 
Instructors  /  Chief  of 
Training  /  Unit  Leader 

Quality 
Assurance  / 
Staff  &  Faculty 

Sustain  ( n  =  26) 

10 

8 

8 

Execute  ( n  =28) 

8 

11 

9 

Plan/Dev  ( n  =  18) 

5 

7 

6 

Total  (N  =72) 

23 

26 

23 

Questions  were  asked  about  the  STTA/ALM  training  which  was  sponsored  by  TRADOC. 
Table  6  presents  the  numbers  of  individuals  who  attended  at  least  one  SITA  course  and  those 
who  did  not  attend  any  course  or  for  whom  no  data  were  available.  For  course  managers, 
training  developers  and  instructors,  the  primary  individuals  responsible  for  the  course  changes, 
the  percentage  of  SITA  attendance  was  highest  for  courses  in  the  Execute  stage,  then  Sustain, 
and  lastly  Plan/Dev.  The  low  percentages  of  individuals  in  the  Plan/Dev  stage  who  received 
SITA  training  may  have  inhibited  their  ability  to  execute  change. 
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Table  6 

SITA  Attendance  by  Innovation  Stage  and  Duty  Position 


Innovation  Stage 

N  of 

Individuals 

Attended  SITA 

n  (%) 

Not  Attended  SITA 
or  Not  Asked 
n  (%) 

Training  Developers  /  Course  Manager  /  Instructors 

Sustain 

18 

10  (56%) 

8  (44%) 

Execute 

19 

16  (84%) 

3  (16%) 

Plan/Dev 

12 

4  (33%) 

8  (67%) 

Total 

49 

30(61%) 

19  (39%) 

Staff  and  Faculty,  and  Quality  Assurance 

Sustain 

8 

7  (87%) 

1  (13%) 

Execute 

9 

4  (45%) 

5  (55%) 

Plan/Dev 

6 

4  (67%) 

2  (33%) 

Total 

23 

15  (65%) 

8  (35%) 

All  Participants 

Sustain 

26 

17  (65%) 

9  (35%) 

Execute 

28 

20  (71%) 

8  (29%) 

Plan/Dev 

18 

8  (44%) 

10  (56%) 

Total 

72 

45  (63%) 

27  (37%) 

Note.  Percentages  based  on  the  number  of  individuals  in  each  innovation  stage.  SITA 
attendance  based  on  attendance  in  at  least  one  SITA  course.  Individuals  who  attended  more  than 
one  course  were  counted  only  once. 


Timelines 

The  cumulative  time  to  make  changes  to  a  course  and  to  implement  it  was  used  to 
document  the  rate  at  which  an  innovation  was  adopted.  However,  due  to  the  turnover  in 
personnel  from  the  start  of  the  SITA  effort  to  the  time  individuals  were  interviewed,  in  some 
courses  there  were  information  gaps.  Thus,  this  section  of  the  report  simply  summarizes  the 
timelines  by  course  status.  It  is  noted  that  timelines  can  vary  because  of  course  length,  e.g., 
pilots  or  iterations  of  a  course  are  faster  with  short  courses  or  with  a  pilot  which  involves  only  a 
section  of  a  course.  Time  was  also  somewhat  dependent  on  the  dates  when  the  interviews  were 
collected  over  the  August  2014  to  March  2015  research  period. 

For  each  course,  the  start  point  was  when  planning  began  and  the  end  point  was  when  the 
data  were  collected.  Planning  efforts  typically  started  three  to  six  months  after  the  course 
decision  was  made,  although  in  some  instances  the  exact  decision  date  was  not  known  to  those 
who  were  interviewed.  In  general,  planning  started  around  May  2013  with  the  bulk  of  the 
interviews  completed  by  the  end  of  November  2014,  a  period  of  20  months.  Consequently,  the 
timelines  were  typically  based  on  a  20-month  window  (there  were  exceptions,  as  some  courses 
started  planning  earlier  and/or  were  interviewed  later). 
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Table  7  summarizes  the  status  of  the  courses  by  innovation  stage  and  the  time  required  to 
progress  to  their  respective  stages.  The  total  times  for  courses  in  the  Sustain  and  Execute  stages 
were  very  similar,  but  courses  in  the  Execute  stage  had  not  progressed  as  far  as  those  in  the 
Sustain  stage.  Delays  occurred  for  each  course  in  the  Execute  stage.  These  delays  meant  that  the 
first  implementation  or  pilot  for  courses  in  the  Execute  stage  was  6  to  12  months  later  than  what 
occurred  with  courses  in  the  Sustain  stage.  For  the  three  courses  in  the  Plan/Dev  stage,  none  had 
been  executed  as  desired,  with  the  timelines  ranging  from  12  to  21  months. 

Based  on  the  Sustain  courses  timelines,  the  time  required  from  the  initial  decision  to 
make  course  changes  and  ensure  the  changes  are  working  would  be  1.5  to  2  years.  This  time 
could  be  shorter  if  no  external  factors  influenced  the  process  and  manpower  was  satisfactory. 


Table  7 

Cumulative  Time  From  Start  of  Planning  to  Course  Status  When  Data  Were  Collected 


Innovation 

Stage 

Description  of  Course  Status 

Time  in 
months 

Reasons  for  Delays 

Sustain 

-  All  three  courses  were  implemented  at 
least  three  times. 

-  Personnel  indicated  that  no  major 
changes  envisioned;  had  achieved  a 
steady  state. 

11  to  17 

Execute 

-  Two  courses  were  implemented  twice. 

-  Two  courses  were  implemented  once. 

-  Changes  were  ongoing  for  each  course. 

10  to  16 

Substantial  doctrinal  changes 
required  before  ALM  concepts 
could  be  incorporated,  first  pilot 
indicated  the  approach  did  not 
work  and  revisions  were 
required,  and/or  a  formal 
decision  to  stagger  the 
implementation  of  ALM 
concepts  in  courses  of  interest. 

Plan/Dev 

-  No  course  had  been  implemented. 

12  to  21 

Training  developer  overload 
which  prevented  the  training 
development  work  from  being 
completed,  a  major  restructure  of 
the  sequence  of  the  course  which 
delayed  implementation  of  ALM, 
and/or  formal  decision  to  stagger 
the  implementation  of  ALM. 
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Perceived  Status  of  Decision-making,  Planning,  and  Developing  Stages 

Background  questionnaire.  Table  8  indicates  the  progress  made  in  the  different  course 
categories  regarding  planning  and  development.  The  responses  to  these  specific  items  on  the 
background  questionnaire  reflect  the  overall  differences  in  progress  among  the  courses  in  the 
three  innovation  stages. 

Table  8 

Percentage  of  Individuals  Responding  to  Background  Questions  on  Progress  of  Major  Revisions 
(most  frequent  response  for  each  innovation  stage  is  in  boldface ) 


Topic 

Rating  Scale 

Sustain 

Innovation  Stage 

Execute  Plan/Dev 

Identify 

Not  identified 

— 

12% 

12% 

Revisions 

Somewhat  identified  &  agreed  upon 

— 

63% 

63% 

Fully  identified  &  agreed  upon 

100% 

25% 

25% 

Total  number  of  respondents 

6 

8 

8 

Identify 

Not  agreed  upon 

___ 

24% 

— 

Solutions 

Somewhat  agreed  upon 

— 

38% 

86% 

Fully  agreed  upon 

100% 

38% 

14% 

Total  number  of  respondents 

6 

8 

7 

Develop 

Not  developed 

— 

33% 

83% 

Course 

Draft  materials 

17% 

11% 

17% 

Materials 

Finalized  materials 

83% 

55% 

— 

Total  number  of  respondents 

6 

9 

6 

Develop 

Not  developed 

33% 

33% 

40% 

Instructor 

Draft  materials 

— 

22% 

60% 

Training 

Fully  developed  &  applied 

67% 

44% 

— 

Materials 

Total  number  of  respondents 

3a 

9 

5 

a  One  individual’s  response  to  this  question  was  ambiguous  and  not  included  in  the  final  tally. 
Due  to  the  small  number  of  responses,  it  is  difficult  to  draw  conclusions  regarding  this  topic  for 
all  courses  in  the  Sustain  stage. 


Note.  Number  of  respondents  to  each  topic  corresponds  to  the  number  of  individuals  who  had 
sufficient  knowledge  to  answer  the  questions.  If  the  respondents  were  unaware  of  the  process, 
they  were  told  not  to  answer  the  question.  A  total  of  45  individuals  responded  to  the  background 
questionnaire,  but  not  all  answered  the  questions  referenced  in  this  table. 


In  summary,  the  ratings  for  the  Sustain  courses  were  distinct  when  compared  to  courses 
in  the  Execute  and  Plan/Dev  stages,  in  that  the  Sustain  ratings  consistently  indicated  the  greatest 
progress  in  identifying  revisions,  identifying  solutions  and  developing  course  materials  (also  see 
Appendix  D).  Of  note  is  that  instructor  materials  had  ratings  of  not  fully  developed  for  every 
course  stage.  Lastly,  no  participant  indicated  that  the  candidate  course  for  the  SIT  A  effort  was 
hard  to  identify. 
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Planning  and  development  questionnaires.  The  answers  to  the  planning  questionnaire 
reflected  the  planners’  expectations,  while  the  answers  to  the  development  questionnaire 
reflected  what  training  developers  actually  experienced.  Tabulation  of  the  ratings  given  to  the 
common  set  of  seven  change  dimensions  in  the  planning  and  development  questionnaires  is  in 
Table  9;  complete  data  are  in  Appendix  D. 

Trends  from  the  planning  results.  There  were  no  major  distinctions  among  course 
stages  regarding  expected  changes. 

•  Major  changes  were  typically  expected,  e.g.,  re-write  of  the  program  of 
instruction  and  lesson  plans  plus  extensive  instructor  training,  and  no  one 
expected  making  the  changes  to  be  “simple.” 

•  Difficulty  of  change  was  rated  as  moderate  or  extremely  difficult 

•  Changes  in  instructor  behavior  were  viewed  as  somewhat  difficult 

•  Time  required  was  anticipated  as  being  extreme 

•  Changes  were  expected  to  be  done  with  current  personnel 

•  Moderate  increase  in  workload  was  expected 

•  Costs  were  anticipated  to  be  inexpensive  or  somewhat  costly. 

Development  results.  Three  major  trends  occurred  in  the  ratings. 

•  Compared  to  the  planning  estimates,  the  training  developers’  responses  were 
more  varied.  This  greater  spread  reflected  in  part  the  unique  characteristics  and 
dynamics  of  each  course  in  a  given  stage. 

•  The  number  of  change  dimensions  with  the  most  challenging  rating  varied 
systematically  with  course  stage.  Sustain  courses  had  the  fewest  challenging 
ratings  (2  of  7),  followed  by  Plan/Dev  courses  (4  of  7),  with  Execute  courses 
having  the  most  challenging  ratings  (6  of  7).  Also  the  ratings  were  most  diverse 
for  the  courses  in  the  Execute  phase,  which  could  reflect  the  larger  number  of 
courses  in  this  phase. 

•  Overall,  the  development  responses  for  the  Sustain  courses  reflected  minor  to 
moderate  perceived  changes  in  time  required,  manpower  required,  workload,  and 
costs.  In  contrast,  responses  for  the  Execute  and  Plan/Dev  courses  indicated  more 
time  required  and  greater  workload.  These  ratings  were  also  reflected  in  the 
interviews. 

Other  trends.  Not  all  changes  were  judged  as  “large-scale”  during  development, 
although  “large  scale”  changes  were  anticipated  for  all  courses  in  the  planning  stage.  Training 
developers  agreed  with  the  planning  estimates  as  the  changes  were  not  judged  as  “simple”  to 
accomplish.  Ratings  for  changes  in  instructor  behavior  were  diverse  for  courses  in  the  Sustain 
stage,  and  less  variable  in  the  other  two  stages.  Some  additional  manpower  was  required  in  some 
Sustain  and  Execute  courses;  no  additional  manpower  was  used  for  the  Plan/Dev  courses.  These 
responses  agree  with  the  manpower  numbers  presented  in  the  next  section.  Lastly,  there  were  no 
costs  associated  with  new  or  revised  training  technologies  in  any  of  the  courses.  The  costs  cited 
for  the  Execute  courses  reflect  personnel  costs  associated  with  contractor  support  to  the  CoE. 
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Table  9 

Percentage  of  Individuals  Responding  to  Questions  on  Change  Dimensions:  Planning  and 
Development  Stages  (most  frequent  response  is  in  boldface ) 


Change 

Dimension 

Rating  Scale 

Planning 

Questionnaire 

Development  Questionnaire 

All  Courses 
(n  =  10) 

Sustain 
(n  =  7) 

Execute 
(n  =  10) 

Plan/Dev 
(n  =  6) 

Degree  of 

Minor 

— 

— 

30% 

17% 

Change 

Moderate 

20% 

57% 

20% 

50% 

Large-scale 

80% 

43% 

50% 

33% 

Difficulty  of 

Simple 

_ 

14% 

20% 

17% 

Change 

Moderate 

50% 

72% 

50% 

33% 

Extremely  difficult 

50% 

14% 

30% 

50% 

Changes  in 

Minor 

_ 

43% 

20% 

17% 

Instructor 

Somewhat  difficult 

70% 

28% 

40% 

66% 

Behavior 

Substantial 

30% 

28% 

30% 

17% 

Time 

No  additional 

_ 

_ 

20% 

_ 

Required 

Moderate 

20% 

71% 

20% 

50% 

Extreme 

80% 

29% 

60% 

50% 

Manpower 

Current  personnel 

50% 

29% 

30% 

67% 

Required 

Some  additional 

20% 

71% 

30% 

— 

Much  additional 

30% 

— 

40% 

33% 

Workload 

The  same 

10% 

29% 

10% 

17% 

Moderate  increase 

70% 

71% 

50% 

17% 

Substantial  increase 

20% 

— 

50% 

66% 

Costs 

Inexpensive 

40% 

100% 

20% 

66% 

Somewhat  costly 

30% 

— 

30% 

33% 

Very  costly 

10% 

— 

50% 

— 

Notes.  For  the  planning  questionnaire,  all  courses  are  grouped  together  because  of  the 
consistency  in  responses  and  the  limited  number  of  responses.  . 
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Manpower  Required 

The  data  on  the  typical  number  of  training  developers  required  to  develop  the  materials 
was  extremely  limited.  Responses  were  available  for  only  six  of  the  ten  courses.  Considering  all 
six  courses,  participants  indicated  there  were  either  one  or  two  full-time  training  developers.  For 
five  of  the  six  courses  (only  Execute  and  Sustain),  part-time  training  developers  were  available. 

Factors  that  Affected  the  Rate  of  Change  and  Extent  of  Adoption  of  the  ALM 

This  section  presents  results  on  the  major  factors  that  affected  the  adoption  of  the  ALM. 
This  includes  Rogers’  characteristics  as  well  as  other  factors.  The  major  factors  that  impacted 
the  rate  of  change  for  courses  in  each  of  the  innovation  stages  is  shown  in  Figure  3.  Facilitating 
factors  are  preceded  by  a  plus  (+)  sign;  inhibiting  factors  are  preceded  by  a  minus  (-)  sign. 


Sustain  Execute  Plan/Dev 


+  Well-planned:  Initial 
agreement  on  revisions 

+  Pilots  w  observations 
which  created  postiive 
feedback  loop  between  TD 
and  instructors 


+  Minimal  paradigm  shift 


+/-  Consistent  attention  to 
ensuring  instructor 
implementation  when 
instructors  had  difficulty 
making  changes 


-  Increased  TD  and 
instructor  workload 


+  At  least  one  pilot 
+  Less  PowerPoint 

+  High  %  of  TDs  and 
Instructors  received  ALM 
training 

-Higher-level  factors 
required  change  (not  ALM); 
caused  delays 


-  Increased  training 
developer  and  instructor 
workload 


-  Coordination  between 
TDs  and  instructors 
delayed  implementation 


+  Less  PowerPoint 


-  Increased  TD  and 
instructor  workload 

-  Shortage  of  TDs  and 

instructors 

-  Some  organizational 
barriers  to  effective 

TD/instructor  interaction 


-  Higher-level  factors 
required  change  (not  ALM) 
-  caused  delays 


Figure  3.  Major  factors  that  impacted  the  rate  of  change  for  courses  in  each  innovation  stage. 
(“TD”  stands  for  “training  developer.”) 


18 


The  major  trends  and  themes  across  the  courses  and  innovation  stages  are  presented  in 
the  following  order: 

•  Instructor  knowledge  and  understanding  of  ALM 

•  Interactions  among  key  groups  of  personnel 

•  Personnel  issues,  primarily  turbulence  and  shortages 

•  Course  champion 

•  Roger’s  rate  of  adoption  characteristics 

Instructor  knowledge  and  understanding  of  ALM.  Most  comments  on  instructors 
came  from  participants  in  the  Sustain  and  Execute  stages,  as  these  courses  had  actually  been 
executed.  Four  themes  emerged  regarding  how  instructors  influence  the  innovation  process. 

Theme  one.  Extent  to  which  instructors  knew  or  needed  to  know  what  ALM  meant. 

Not  all  instructors  had  attended  ALM  classes,  and  not  all  felt  that  the  instruction  received 
was  sufficient.  A  trend  in  the  data  indicated  that  a  large  portion  of  instructors  did  not 
know  the  intent  or  purpose  of  ALM.  Responses  from  both  instructors  and  training 
developers  indicated  a  need  for  instructors  to  receive  much  more  instruction  on  what 
ALM  is,  how  to  implement  ALM,  and  how  to  develop  instructional  materials  to  execute 
ALM.  It  was  felt  that  this  instructor  training  should  be  similar  to  what  training 
developers  receive  and  should  be  executed  prior  to  teaching  the  course. 

Theme  two.  Degree  to  which  instructors  could  actually  implement  ALM  concepts/ 
principles  /  techniques,  primarily  being  able  to  facilitate  discussions  and  implement  a 
learner-centric  instructional  approach.  To  actually  implement  and  execute  ALM,  the 
participants  indicated  that  more  than  knowledge  of  ALM  was  needed.  An  instructor  must 
also  be  a  subject  matter  expert;  must  prepare  for  what  is  not  in  the  lesson  plan  in  order  to 
answer  student  questions;  and  must  establish  an  open-line  of  communication  with 
students.  The  ability  to  facilitate  seemed  particularly  challenging  as  indicated  by 
comments  regarding  the  difficulty  in  changing  to  a  facilitation  and  learner-center  mode 
for  instructors  who  had  been  instructors  for  a  long  time.  Some  instructors  were  reported 
as  reverting  back  to  the  legacy  style  and  in  need  of  constant  reminding  to  be  a  facilitator 
and  learner-centered.  For  courses  that  had  several  iterations,  respondents  commented 
that  instructors  became  more  proficient  with  time,  but  it  was  often  difficult  for  instructors 
to  give  up  control  and  function  as  a  facilitator. 

Theme  three.  Subject  matter  expertise.  Some  comments  stressed  that  instructors  must 
be  extremely  knowledgeable  regarding  technical  content  to  be  facilitators  and  good 
mentors.  They  needed  to  know  material  beyond  the  scope  of  the  course  content  per  se  in 
order  to  address  student  questions.  In  addition  to  ALM  training,  some  instructors 
indicated  they  will  continue  the  policy  of  having  new  instructors  shadow  courses  or 
pairing  new  instructors  with  current  instructors  (right-seat-ride),  and  certifying  them  prior 
to  allowing  them  teach  independently. 

Theme  four.  Concern  about  sustaining  instructor  expertise.  This  concern  was  cited 
primarily  by  individuals  associated  with  courses  with  military  instructors  where  instructor 
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turnover  was  high  (e.g.,  every  18  months).  Train-up  time  could  be  long,  and  once 
instructors  become  proficient  in  their  role,  it  was  time  for  them  to  leave.  The  issues 
raised  in  the  current  effort  regarding  instructor  expertise  in  ALM  could  be  exacerbated  in 
the  future  when  there  may  be  less  focus  on  ALM  concepts  per  se,  resulting  in  a 
degradation  in  the  knowledge  of  ALM  and  techniques  necessary  to  enable  instructors  to 
implement  these  concepts  with  skill  and  understanding. 

Interactions  among  key  units/groups.  The  nature  and  timing  of  the  interactions  and 
lines  of  communication  between  key  personnel  and  units  also  emerged  as  critical  to  ALM 
implementation.  Three  themes  emerged. 

Theme  one.  Criticality  of  coordination  between  training  developers  and  instructors  and 
to  establish  such  coordination  early  in  the  innovation  process.  This  was  a  major  issue. 
For  example,  the  important  of  coordination  was  stressed  in  one  course  in  the  Sustain 
stage  because  it  was  hard  to  get  the  instructors  and  the  training  developers  at  the  same 
level  of  understanding.  This  was  complicated  by  the  fact  that  instructors  had  varying 
levels  of  expertise  and/or  that  some  were  civilian  and  others  were  military.  The  initial 
lack  of  coordination  between  training  developers  and  instructors  delayed  the  process,  but 
with  repetitions  of  the  course  instructors  eventually  understood  the  training  developers’ 
intent.  The  general  consensus  was  that  if  the  two  groups  had  collaborated  initially,  the 
process  would  have  been  much  smoother. 

Theme  two.  Coordination  among  all  key  players.  The  broader  issue  of  coordination 
was  reflected  in  comments  by  some  participants  in  the  Execute  stage.  Instructors  in  one 
course  thought  the  decision-making  process  should  have  been  different.  Specifically, 
they  said  the  first  step  should  have  been  ALM  training  for  everyone  (e.g.,  instructors, 
training  developers,  course  managers),  then  course  selection,  then  everyone  should  plan, 
and  finally  the  training  developers  and  instructors  should  work  together  on  changes.  That 
would  have  reduced  the  disconnects  and  misunderstandings,  which  happened  when  the 
information  was  simply  handed  over  to  the  instructors. 

Theme  three.  Establishing  communication  when  key  players  are  under  different  lines 
of  authority.  Coordination  problems  were  stressed  in  one  course  in  the  Plan/Dev  stage, 
where  the  lines  of  authority  for  instructors  and  training  developers  differed.  This 
inhibited  interactions  and  communication/coordination  between  these  two  groups  of 
personnel,  which  in  turn  inhibited  appropriate  and  timely  development  of  the  revised 
lessons.  Coordination  was  also  delayed  for  a  course  in  the  Sustain  stage,  where  the 
training  developers  and  instructors  were  under  different  commands. 

Patterns  of  interactions .  Who  actually  interacted  with  each  other?  One  question  on  the 
background  questionnaire  asked  participants  to  indicate  their  primary  interactions  between 
themselves  and  other  units  and  whether  the  interaction  was  two-way  or  one-way.  The  results 
presented  here  are  only  for  the  four  major  groups  involved  in  the  actual  course  conversion 
process:  course  managers,  training  developers,  instructors,  and  directors/commanders.  These 
interactions  are  shown  in  Figure  4  and  summarized  after  the  figure. 
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The  3  Sustain  Courses 


•  Solid  two-headed  arrow  indicates 
two-way  interaction 

•  Dashed  one-headed  arrow  indicates 
one-way  interaction 

•  Circle  represents  interaction  within 
a  group 

•  #  represents  number  of  courses 
which  had  a  specific  form  of 
interaction 


The  4  Execute  Courses 


The  3  Dev/ PI  an  Courses 


Figure  4.  Interactions  between  key  groups  of  personnel. 
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Figure  2.  Interactions  between  training  developers  and  instructors.  Two-way 
interactions  existed  between  training  developers  and  instructors  for  each  course  except  one  in  the 
Plan/Dev  stage,  where  training  developers  interacted  with  instructors  but  not  vice  versa. 
However,  some  training  developers  and  some  instructors  indicated  their  primary  interactions 
were  within  their  own  group.  It  is  noted  that  one  reason  for  the  circle  around  instructors 
(indicating  primary  interactions  within  this  group)  came  from  courses  where  the  instructors  were 
also  the  course  writers. 

Figure  2.  Interactions  between  training  developers  and  course  managers.  This  link  is 
probably  a  function  of  the  organizational  structure  within  most  CoEs.  Training  developers  and 
course  managers  typically  had  two-way-interactions.  When  one-way  interactions  occurred,  they 
reflected  training  developers  initiating  interactions  with  other  groups.  Also  training  developers 
typically  initiated  interactions  with  the  director  or  commander. 

There  are  limitations  to  these  interaction  data.  First,  is  that  although  participants  were 
asked  to  focus  on  primary  interactions,  it  is  likely  that  many  simply  indicated  if  they  interacted 
with  another  group.  Second,  these  results  do  not  indicate  changes  over  time,  which  were  often 
described  by  participants  when  they  were  interviewed.  Third,  generalizations  beyond  this 
sample  regarding  typical  interactions  for  courses  at  different  stages  of  innovation  are  limited  due 
to  the  small  sample  size. 

Personnel  issues.  Most  personnel  issues  focused  on  turbulence,  longevity,  shortages, 
and  instructor  selection. 

Primary  theme.  Personnel  turbulence,  longevity,  and  shortages  interacted  with  each 
other  to  delay  the  innovation  process.  Instructor  turbulence  and  shortages  occurred  in 
all  course  stages.  In  one  case  in  the  Sustain  stage,  participants  indicated  some  instructors 
in  the  initial  pilot  were  already  gone,  which  impacted  sustainment  of  the  changes. 
Participants  in  the  Plan/Dev  stage  indicated  there  was  often  insufficient  time  or  training 
to  adequately  prepare  instructors.  Overall,  the  general  response  by  participants  was  that 
these  factors  meant  that  instructor  preparation  must  continue.  Turbulence  in  other 
positions  also  inhibited  the  process;  specifically  cited  was  turbulence  in  the  course 
manager  position  ( Sustain  stage). 

Other  themes.  Instructor  selection,  leadership  support,  and  time  pressures  on  training 
developers.  These  themes  were  not  common  across  course  stages,  but  do  reflect  areas 
cited  by  some  participants  that  impacted  innovation.  Instructor  selection,  that  is  selecting 
or  finding  the  right  individuals  who  can  implement  ALM  techniques  and  will  invest  the 
necessary  time  as  instructor  to  maximize  these  techniques,  was  cited  as  a  concern 
(Sustain  stage).  Leadership  support  was  viewed  as  essential,  as  support  for  the  ALM 
changes  were  more  widely  accepted  when  the  leadership  changed  (Sustain  stage).  In 
both  the  Sustain  and  Execute  stages,  some  training  developers  indicated  that  time 
pressures  meant  they  were  more  reactive  than  proactive,  and  needed  more  time  for 
planning  and  analysis  to  implement  good  revisions. 


22 


Champion.  In  general,  champions  were  not  cited  in  the  questionnaires  or  the  interviews. 
From  the  interview  responses,  it  was  clear  that  for  two  of  the  Sustain  courses  and  one  Plan/Dev 
course  specific  individuals  were  the  driving  force  behind  the  changes.  For  the  Sustain  courses, 
two  individuals,  a  training  developer  and  an  instructor,  were  present  throughout  the  innovation 
process  and  had  major  responsibilities  in  planning  and  developing  course  changes  as  well  as 
implementing  them.  For  one  Plan/Dev  course,  a  training  developer  was  identified  as 
instrumental  in  planning  the  changes  and  in  establishing  good  working  relationships  with  the 
instructors  and  course  manager  to  prepare  for  ALM  implementation.  However,  turbulence  in 
this  position  meant  that  this  individual  did  not  remain  throughout  the  ALM  process,  and  thus  did 
not  entirely  meet  Rogers’  definition  of  a  champion.  In  fact,  turbulence  in  instructor/  training 
developer  /  course  manager  positions,  in  general,  may  have  inhibited  individuals  serving  as  a 
“champion.” 

Roger’s  characteristics  that  affect  rate  of  adoption.  The  interviews  were  coded  for 
comments  that  corresponded  to  the  rate  of  adoption  characteristics  in  Rogers’  (2003)  work. 
Complexity  clearly  inhibited  the  rate  of  adoption  for  all  courses.  Although  trialability, 
observability,  and  compatibility  can  have  positive  impacts  on  adoption,  there  were  mixed  results 
across  the  course  stages.  Table  10  summarizes  these  impacts.  Details  for  each  characteristic  are 
in  the  text.  As  pointed  out  in  the  discussion,  these  characteristics  were  not  always  independent. 

Relative  advantage.  Did  those  interviewed  perceive  ALM  concepts  and  approaches  as 
better  than  the  legacy  approach  to  instruction  ?  Participants  from  every  course  perceived 
advantages  from  applying  ALM.  The  primary  advantages  were  as  follows. 

Increase  in  student  motivation.  Instructors  indicated  that  the  revised  course  changed 
student  motivation  and  involvement  mainly  due  to  the  reduction  in  PowerPoint  slides  and 
increase  in  class  discussions.  Responses  also  reflected  that  students  liked  being  in  groups  mainly 
because  they  could  help  each  other  leam  by  conveying  the  information  in  a  different  way  and  by 
staying  engaged  in  the  learning  process. 

AIM  was  a  better  instructional  approach.  Others  indicated  they  thought  the  revised 
hands-on  approach,  an  emphasis  upon  practice  exercises,  holding  students  more  accountable  for 
their  learning,  and  /  or  small-group  work  were  better  approaches.  Instructors  and  course 
managers  from  some  courses  stressed  the  advantages  of  giving  students  more  opportunities  to 
practice  what  they  learned,  especially  with  small  groups.  It  was  noted  that  small  group 
interaction  enabled  problem-solving  at  the  group  level. 

Positive  reaction  to  increased  student  accountability.  Responses  by  some  instructors 
indicated  that  the  increased  accountability  placed  on  students  (e.g.,  homework,  read-aheads) 
allowed  the  instructors  to  more  easily  distinguish  between  students  who  wanted  to  leam  versus 
those  who  were  not  adapting  or  not  ready  to  apply  what  they  learned. 

Other  indices  of  relative  advantage  were  reflected  in  distinct  comments  made  by  those 
interviewed  in  three  courses.  One  was  that  instructors  did  not  want  to  go  back  to  the  old  way  of 
teaching.  Another  was  that  some  prior  graduates  said  they  wanted  to  come  back  and  teach  the 
course.  One  course  leader  had  feedback  from  the  field  that  graduates  were  better  prepared  for 
their  duty  position. 
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Table  10 

Course  Profiles  on  Rogers  ’  Rate  of  Adoption  Characteristics 


Stage  of  Innovation 

Characteristic  Sustain  Execute  Plan/Dev 


Relative  Advantage 

Perceived  positive 
impact  for  all 


•  No  differences  in  courses: 

•  Participants  from  every  course  perceived  advantages  from  ALM 

•  However,  technical  skills  and/or  procedural  subject  matter  viewed  as  best 
taught  via  legacy,  direct  instruction  methods 


Trialability 

Mixed  impacts 


•  Leveraged  trialability 

•  Minimum  of  3  pilots  in 
each  course 

•  Trials  or  pilots  were 
planned 

•  Systematic 
observations  of  revised 
course 


•  Some  trialability  • 

•  One  or  two  pilots  in  • 

each  course 

•  Pilots  not  planned  • 

•  No  trials  on 
portions  of  a  course 

•  No  planned 
observations 


No  trialability 
Trials  or  pilots 
did  not  occur 
Courses  not 
executed 


Observability 

Mixed  results 


•  Observations  influenced  • 
course  improvements 


Observations 
influenced  course 
improvements  (2 
courses) 

Feedback  on  initial 
implementation 
planned  to  apply  to 
next  iteration  (2 
courses) 


•  No  observations 


Re-invention 


•  Re-invention  of  the  initial  course  implementation  occurred  or  was 
anticipated  across  all  stages. 


Compatibility  • 

Mixed  results 


Transfer  of  some  ALM 
techniques  used  in  other 
courses 

ALM  techniques 
perceived  as  way  to 
address  a  training  gap  in 
one  course 


•  Problem-solving 
exercises  common 
in  some  legacy 
courses 

•  Paradigm  shift  for 
some  courses 


•  Little 

compatibility; 
paradigm  shift  in 
instructional 
techniques 


24 


Stage  of  Innovation 

Characteristic 

Sustain 

Execute 

Plan/Dev 

Complexity 

•  Broad  scope  of  work  for 

•  Broad  scope  of 

•  Shortage  of 

training  developers 

work  for  training 

training 

Slowed  rate  of 

•  How  to  design  exercises 

developers 

developers  for 

adoption 

in  accordance  with 

•  How  to  design 

scope  of  work, 

ALM  concepts 

exercises  in 

with  increased 

Primary  factors 

•  Challenges  for  training 

accordance  with 

workload 

varied  with  course 

developers  and 

ALM  concepts 

•  How  to  design 

instructors  when 

•  Challenges  for 

exercises  so 

students’  backgrounds 

training  developers 

instructors 

were  diverse 

and  instructors  when 

understood 

•  More  instructor 

students’ 

intent 

preparation  time  when 

backgrounds  were 

•  Challenges  for 

applying  ALM 

diverse 

training 

•  High  level  of  subject 

•  More  instructor 

developers  and 

matter  expertise 

preparation  time 

instructors  when 

required  by  instructor 

when  applying 

students’ 

•  Training  developer  and 

ALM 

backgrounds 

instructor  workloads 

•  Higher-order, 

were  diverse 

increased 

external 

•  Lack  of  clarity 

requirements 

regarding 

extended  the 

responsibilities 

development 
process  or  made  it 
more  complex 

of  personnel 

•  Training  developer 

and  instructor 

workloads 

increased 

The  primary  area  where  ALM  techniques/approaches  were  not  perceived  as  having  a 
relative  advantage  was  with  technical  subject  matter.  ALM  techniques/approaches  were  not 
necessarily  perceived  as  having  a  “relative  advantage”  for  technical  and/or  procedural  subject 
matter.  With  technical  content  which  may  impact  safety  or  accuracy  or  is  highly  detailed, 
instructors  felt  that  facilitation  techniques  and/or  small  group  techniques  were  not  applicable 
(, Sustain  and  Execute  stages).  In  other  words,  the  legacy  approach  with  direct  instruction  was 
more  appropriate. 
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Trialability.  How  did  trialability  (course  pilots)  impact  the  innovation  process  and 
course  implementation  ?  Trialability  comments  occurred  only  from  the  Sustain  and  Execute 
courses,  not  from  the  Plan/Dev  courses  as  no  pilots  were  conducted  with  those  courses  (see 
Table  12).  A  summary  of  the  comments  from  the  Sustain  and  Execute  courses  follows. 

Sustcdn  stage.  Pilot  implementations  were  executed  for  each  course  in  the  Sustain  phase. 
In  fact,  the  leadership  for  these  courses  specifically  planned  for  pilots  and  for  systematic 
observations  of  the  pilots  (3  to  5  pilots  were  executed).  This  allowed  instructors  and  training 
developers  to  determine  what  actually  worked  with  students,  and  the  process  created  an 
important  feedback  loop  between  instructors  and  training  developers.  In  the  first  pilot  for  each 
course  in  the  Sustain  stage,  problems  with  the  revised  course  were  identified.  A  synopsis  of 
what  the  participants  said  about  each  Sustain  course  is  provided  below  to  illustrate  the  variations 
as  well  as  the  commonalities  in  how  pilots  were  conducted. 

In  one  Sustain  course,  three  pilot  classes  were  authorized,  and  the  instructors  indicated 
that  three  pilots  were  necessary  to  determine  the  best  approach.  During  the  pilots,  an  assistant 
instructor  took  notes;  changes  were  made  in  the  next  class. 

In  another  Sustain  course,  five  pilots  were  executed  and  the  first  did  not  work,  so  the 
focus  changed.  Each  pilot  was  implemented  on  a  specific  lesson  (not  the  entire  course)  with 
small  groups.  After  each  pilot,  the  lesson  was  examined  and  refined,  and  this  cycle  was  repeated 
with  another  pilot  lesson.  Each  time  issues  were  identified  and  changes  were  made.  A  student 
survey  was  also  given  after  one  of  the  pilots.  The  leadership  found  it  was  necessary  to  work  with 
the  instructors  to  explain  the  changes  and  their  new  role. 

In  third  Sustain  course,  training  developers  or  their  representatives  observed  the  classes. 
The  first  revision  of  the  course  was  found  to  be  too  difficult  for  students.  In  the  second  version, 
the  content  was  broken  down,  the  course  was  made  more  student-centric,  and  the  instructors 
acted  more  as  facilitators.  Those  interviewed  found  instructor  resistance  at  first  which 
diminished  with  the  later  pilots.  Training  developers  also  learned  they  had  to  rewrite  the  lesson 
plans  to  make  them  the  more  specific  on  what  students  needed  to  do.  Thus,  the  revised  lessons 
contained  more  notes  for  the  instructors.  After  three  iterations,  participants  indicated  that  no 
major  changes  were  expected. 

Execute  stage.  For  the  courses  in  the  Execute  phase,  pilots  occurred,  but  systematic  plans 
for  the  pilots  and  for  observation  of  the  pilots  did  not  exist.  In  addition,  there  was  no  attempt  to 
pilot  specific  lessons  versus  the  entire  course.  Participants  did  indicate  that  they  made  revisions 
based  on  the  first  iteration,  which  provides  additional  support  for  the  value  of  conducting  pilots 
when  revising  courses. 
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Observability.  Could  the  impact  of  the  changes  be  observed,  either  in  the  students  or 
instructors?  With  curriculum  changes,  success  of  an  innovation  can  be  observed  to  some  extent. 
For  example,  instructors  can  be  monitored  to  determine  if  they  are  implementing  the  revised 
lessons  as  planned,  and  students  can  be  observed  to  determine  their  reaction  to  the  instruction. 

Results  on  the  trialability  and  relative  advantage  factors  showed  that  behavioral  changes 
in  students  and  instructors  could  be  and  were  observed.  In  fact,  these  changes  were  often  the 
basis  for  continued  improvements  to  courses  in  both  the  Sustain  and  Execute  stages.  However, 
there  were  no  objective  measures  of  performance  which  compared  student  performance  before 
and  after  course  changes  were  made.  This  topic  is  discussed  more  thoroughly  in  the  section  on 
Student  Outcomes. 

Compatibility.  To  what  extent  were  ALM  concepts  and  objectives  consistent  with  existing 
values,  needs  or  past  experiences  of  course  personnel?  Based  on  the  participants’  comments, 
compatibility  was  not  a  major  factor  affecting  the  rate  of  innovation.  However,  for  four  courses 
it  had  a  positive  impact.  The  reasons  compatibility  had  a  positive  impact  for  these  courses  are 
presented  next. 

For  one  course  in  the  Sustain  stage,  personnel  had  applied  ALM  techniques  such  as 
facilitation  and  group  work  in  another  course.  This  prior  experience  transferred  to  the  course  in 
the  SITA  effort.  For  two  other  courses,  instructors  had  used  some  ALM  techniques  as  well.  For 
once  course  in  the  Sustain  stage,  small  group  and  hands-on  instruction  were  not  new.  One 
course  in  the  Execute  stage  had  incorporated  problem  solving  exercises  in  the  legacy  course. 

For  another  Sustain  course,  personnel  had  recognized  a  training  gap  in  the  course.  The 
course  as  structured  did  not  meet  the  Army’s  needs  and  therefore  changes  like  those  proposed  in 
ALM,  which  stressed  students  being  more  accountable  for  their  learning,  were  perceived  as 
needed.  This  need  motivated  those  involved  with  making  course  changes,  but  it  did  not 
necessarily  provide  lessons  learned  as  was  the  case  when  ALM  techniques  had  been  applied  in 
another  course. 

In  general,  it  appeared  that  for  the  other  courses  a  paradigm  shift  in  most  instructional 
approaches  was  required.  This  topic  is  discussed  further  under  the  re-invention  section. 

Complexity.  How  did  complexity  impact  the  rate  of  adoption  and  what  types  of 
complexity  occurred?  Complexity  of  the  innovation  typically  slows  the  rate  of  adoption.  And 
complexity  played  a  role  in  each  course.  However,  the  nature  of  complexity  varied  with  the 
course.  In  general,  there  was  an  increase  in  workload  for  both  training  developers  and 
instructors  which  delayed  the  rate  of  adoption. 

The  underlying  triggers  for  increased  workload  were  the  very  nature  and  scope  of  the 
training  developers’  and  instructors’  tasks,  that  is,  training  developers  had  to  develop  a  new  or 
revised  program  of  instruction  (POI)  for  an  entire  course  which  could  also  involve  new  content, 
and  instructors  had  to  teach  a  new  or  revised  course  which  typically  required  different 
instructional  techniques.  The  lesson  plans  for  courses  in  the  effort  ranged  from  less  than  20  to  50 
and  to  almost  100  in  one  instance.  The  extent  to  which  ALM  revisions  represented  a  paradigm 
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shift  complicated  both  the  development  and  instructional  processes.  When  the  numbers  of 
training  developer  personnel  did  not  change  or  were  insufficient,  then  the  time  required  to  make 
and  implement  the  revisions  increased.  In  addition,  for  most  courses,  the  front-end  analysis  of 
how  to  revise  the  courses  was  limited,  and  training  developers  often  made  analysis  and 
development  decisions  simultaneously.  Complexity  factors  are  described  in  turn  for  training 
developers  and  instructors. 

Training  developers.  In  the  Sustain  stage,  training  developers  had  to  examine  every 
lesson  plan  and  rewrite  the  lesson  plans  to  accommodate  ALM  concepts.  In  summary,  these 
requirements  contributed  to  increased  workload  in  all  courses  in  the  Sustain  stage.  The  revisions 
for  each  course  were  accomplished  with  assigned  personnel  but  all  training  developers  indicated 
that  considerable  additional  time  had  to  be  devoted  to  the  process.  More  information  on  each 
course  is  provided  next. 

For  one  course,  the  revisions  were  a  major  paradigm  shift  in  training  techniques  and  how 
the  course  was  taught.  Consequently  the  training  developer  revised  one  part  of  the  course,  and 
assessed  its  success  and  feasibility  before  starting  on  the  next  part.  In  addition,  there  was  a 
restricting  and  integration  of  lesson  plans  reducing  them  by  half. 

In  another  course,  although  many  of  the  instructional  techniques  under  development  were 
similar  to  ALM  concepts,  the  pilots  provided  instructional  feedback  from  the  instructors  which 
required  additional  changes  by  the  training  developers.  Also  training  developers  had  to  re¬ 
sequence  the  instructional  models  and  the  course  length  changed.  Another  complicating  factor 
for  training  developers  was  how  to  apply  ALM  concepts  when  the  student  backgrounds  were 
diverse  and  proficiency  levels  varied.  Training  developers  stated  it  was  challenging  to  determine 
how  to  revise  the  course  and  develop  appropriate  exercises  because  sufficient  guidance  on  these 
topics  did  not  occur  until  seven  or  eight  weeks  into  the  course. 

In  the  third  course,  the  instructor  was  also  a  developer/writer.  Although  this  approach 
may  have  allowed  for  better  transfer  of  lesson  plans  to  the  classroom,  it  increased  the  workload 
substantially. 

In  the  Execute  courses,  training  developers  had  the  same  challenges  as  those  in  the 
Sustain  phase.  Complexity  increased  the  timeline  for  course  changes. 

For  three  of  the  four  Execute  courses,  training  developers  had  to  focus  on  more  than 
ALM  revisions.  Developers  had  to  address  higher-order,  external  requirements,  such  as 
doctrinal  changes,  restructuring  of  the  course  content,  change  in  course  length,  and/or  the  need  to 
tailor  the  course  to  the  both  the  active  and  reserve  components  made  the  development  process 
longer  and/or  more  complex.  These  external  requirements  were  the  primary  factors  impacting 
the  longer  timeline  for  the  Execute  versus  the  Sustain  courses. 

In  addition,  in  one  course,  developers  also  indicated  that  it  was  hard  to  apply  ALM 
concepts  to  a  course  which  focused  on  foundational  skills  as  the  students  had  limited  experience. 
Scenarios  had  to  be  very  basic  in  order  to  facilitate  discussions.  Training  developers  in  one 
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course  estimated  their  workload  doubled  or  tripled.  In  another  course,  training  developers 
indicated  that  rubies  were  not  always  easy  to  develop. 

In  the  Plan/clev  stage,  the  main  factors  contributing  to  complexity  were  scenario 
development  and  writing  lesson  plans  so  instructors  could  implement  ALM  concepts 
appropriately. 

In  one  course,  the  training  developer  indicated  it  was  a  challenge  to  ensure  that  the  lesson 
plans  were  written  so  instructors  understood  the  intent  and  how  ALM  concepts  were  to  be 
applied.  Notes  were  added  to  the  lesson  plans  to  assist  the  instructors.  These  difficulties  were 
compounded  by  lack  of  clarity  regarding  the  responsibilities  of  personnel  within  the 
organization.  The  workload  was  so  great  for  the  single  training  developer  that  the  course 
revisions  could  not  be  completed. 

In  the  other  two  courses,  a  primary  challenge  was  developing  scenarios  or  exercises  for 
students  that  would  allow  instructors  to  facilitate  discussions.  This  was  even  more  difficult  when 
the  student  population  was  diverse  in  terms  of  background,  knowledge,  and  skills.  A  training 
developer  indicated  that  often  the  instructor  must  flesh  out  the  details  on  how  the  exercises  could 
best  be  conducted. 

Instructors.  Instructors  in  courses  where  pilots  were  conducted  indicated  they  typically 
revised  what  was  done  on  the  first  pilot  as  they  learned  how  to  apply  ALM  concepts  better.  In 
all  stages  where  the  instructor  was  a  developer/writer,  there  was  an  extensive  amount  of  research 
and  work  to  ensure  that  the  lesson  plans  had  the  necessary  background  information,  that  tests 
were  revised,  and  that  they,  as  instructors,  were  sufficiently  knowledgeable  to  facilitate 
discussions. 

Instructors  in  the  Sustain  stage,  indicated  their  workload  increased;  in  one  case  it  was 
estimated  to  triple  or  quadruple.  Increases  were  attributed  primarily  to  the  requirement  for  an 
instructor  to  be  a  subject  matter  expert  to  facilitate  discussion  and  enable  students  to  be  problem- 
solvers.  A  high-level  of  knowledge  was  needed  to  answer  student  questions,  and  to  correct 
students  when  their  knowledge  was  inaccurate.  Instructor  preparation  time  increased  in  these 
courses.  In  addition,  when  tests  were  modified  to  essay-type  questions  from  multiple-choice 
questions,  instructor  workload  increased.  For  one  course,  instructors  were  challenged  in  how  to 
apply  ALM  concepts  when  the  students’  backgrounds  were  diverse. 

Instructors  in  the  Execute  courses  needed  time  to  adapt  to  ALM  concepts;  they  had  to 
develop  or  obtain  more  source  materials  for  read-aheads  or  home  work;  ALM  required  more 
instructor  preparation.  A  change  from  multiple-choice  tests  to  more  essay  type  tests  required 
more  time  to  grade  as  was  the  case  with  similar  courses  in  the  Sustain  stage.  When  classes  had 
students  who  lacked  prerequisite  knowledge,  instructors  were  challenged  in  addressing  the 
learning  requirements  of  these  students.  One  instructor  indicated  that  this  initially  meant  a 
greater  workload,  but  the  benefit  was  large.  These  factors  all  contributed  to  increased  instructor 
workload. 
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Adequate  instructor  preparation  was  a  theme  in  the  Plan/Dev  stage  as  well.  One 
instructor  indicated  that  preparation  required  individual  initiative,  with  the  instructor  taking  a 
variety  of  courses  in  related  subject  matter  areas  to  be  able  to  address  student  questions.  For 
another  course,  instructors  were  challenged  in  how  to  develop  exercises  for  the  purpose  of 
discussion  and  how  to  adapt  to  individual  differences  in  student’s  prior  knowledge. 

Joint  effects  of  training  developers  and  instructors  on  complexity.  Another  complication 
cited  by  some  participants  from  all  stages  was  that  typically  training  developers  were  civilians 
and  instructors  were  military.  A  leader  from  one  Plan/Dev  course  stressed  this  point.  Instructors 
are  the  subject  matter  experts  on  course  content  and  knowledge/skill  requirements,  while  the 
training  developers  are  skilled  in  writing  lessons  and  assessments.  To  yield  the  desired  product, 
the  two  skill  sets  must  be  merged.  Yet  merging  is  challenging  and  makes  the  development  and 
implementation  processes  more  complex.  In  some  courses,  these  skill  sets  were  merged  by 
having  the  same  individuals  be  both  training  developers  and  instructors,  in  conjunction  with  a 
plan  to  ensure  appropriate  training  for  both  responsibilities. 

Re-invention.  Re-invention  overlapped  with  both  trialability  and  compatibility. 

Relationship  to  compatibility.  As  mentioned  previously  under  compatibility,  some 
courses  in  the  Sustain  stage  had  already  instituted  changes  which  were  perceived  to  be  consistent 
with  ALM  concepts  (facilitation,  group  work,  hands-on  training,  problem-solving  exercises). 
These  were  ALM  techniques  they  were  currently  doing.  However,  they  often  had  to  modify  or 
expand  some  other  instructional  approaches.  Thus,  for  some  CoEs,  ALM  concepts  were  not 
perceived  as  a  major  paradigm  shift  or  as  doing  something  entirely  different. 

Relationship  to  trialability.  Courses  with  more  than  one  pilot  made  revisions  for  the  next 
pilot.  Consequently,  many  of  the  comments  under  trialability  apply  to  re-invention  as  well;  the 
changes  with  the  first  implementation  were  not  fixed,  nor  were  they  necessarily  expected  to  be 
fixed. 


For  example,  in  one  course  in  the  Sustain  stage,  training  developers  stated  that  they  had 
to  rewrite  the  course  two  times,  and  push  back  the  timeline  to  make  the  changes  after  the  first 
pilot.  In  a  course  in  the  Execute  stage,  while  the  training  developer  was  observing,  other  changes 
were  suggested.  Instructors  and  training  developers  both  acknowledged  that  given  the  same 
lesson  plans,  instructors  could  implement  the  lessons  quite  differently.  With  one  course  in  the 
Plan/Dev  stage,  the  training  developer  anticipated  changes  during  execution  even  though  the 
course  had  not  been  implemented. 

Paradigm  shift.  Lastly  for  some  courses,  regardless  of  whether  there  were  pilot  efforts, 
the  initial  revisions  did  involve  a  paradigm  shift  from  the  typical  instructional  techniques  in  the 
legacy  courses.  The  paradigm  shift  was  reflected  primarily  in  switching  to  a  facilitation  mode  of 
instruction  which  required  experienced  instructors  to  change  their  approach  to  the  classroom,  in 
requiring  students  to  be  more  responsible  for  their  own  learning,  in  emphasizing  small  group  or 
peer-to-peer  learning,  and  in  changing  the  nature  of  student  assessments/tests. 
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Student  Outcomes 


Student  outcomes  and  reactions  were  only  available  from  courses  in  the  Sustain  and 
Execute  stages.  Unfortunately,  no  objective  performance  data  were  available  which  compared 
student  performance  in  the  legacy  course  to  performance  in  the  revised  courses.  And  with  some 
courses  in  both  stages,  such  performance  comparisons  were  not  possible  as  the  tests  themselves 
were  modified  to  be  more  consistent  with  ALM  concepts.  Also,  instructors  and  training 
developers  in  two  Sustain  courses  and  one  Execute  course  indicated  it  was  too  early  to  receive 
feedback  on  graduates’  performance  from  the  receiving  units.  Pass  rates  were  not  perceived  as 
changing  in  one  Sustain  course,  while  in  one  Execute  course  pass  rates  declined  from  the  legacy 
course  because  of  increased  course  and  test  rigor. 

Consequently,  feedback  on  student  performance  was  based  primarily  on  the  perceptions 
of  training  developers  and/or  instructors.  Instructors  in  one  Sustain  course  felt  performance  had 
increased,  specifically  citing  better  briefs.  An  instructor  in  the  Execute  stage  could  not  judge 
whether  retention  of  the  subject  matter  in  the  receiving  unit  would  be  better.  For  one  course  in 
the  Sustain  phase,  a  student  was  designated  the  Distinguished  Honor  Graduate  by  the  battalion 
for  receiving  100s  on  all  tests,  an  event  that  was  stated  to  occur  about  once  every  three  years. 

Training  developers  and  instructors  perceived  most  students  as  being  more  engaged  in  all 
Sustain  and  Execute  courses  (e.g.,  asked  more  questions,  more  involved,  more  interested  in 
learning,  showed  more  initiative).  End-of-course  critiques  in  two  Execute  courses  were  positive. 

Instructors  in  two  Execute  courses  indicated  that  some  students  reacted  negatively  (e.g., 
complained  about  homework).  Instructors  attributed  these  reactions  to  the  fact  that  these 
students  did  not  grasp  the  intent  of  the  revised  approach  and  did  not  know  how  to  study. 

ALM  Instructional  Techniques 

Ten  instructional  techniques  were  examined.  Use  of  the  techniques  did  not  discriminate 
course  stage.  Eight  were  clearly  widespread,  being  implemented  in  at  least  80%  of  the  courses. 
In  fact,  four  techniques  were  used  or  planned  to  be  used  in  all  ten  courses.  Table  13  shows  the 
number  of  courses  that  used  ALM  instructional  techniques.  Examples  of  how  these  techniques 
were  implemented  follow,  with  the  most  frequently  used  techniques  cited  first. 

Reduction  in  PowerPoint  slides  and  instructor  lectures.  As  stated,  across  all  courses 
there  was  a  substantial  reduction  in  PowerPoint  slides.  Directly  associated  with  this  was  a 
reduction  in  instructor  lectures.  For  example,  in  one  course,  an  instructor  indicated  there  were 
lessons  with  no  slides.  Yet  PowerPoint  slides  were  retained  for  certain  subjects,  primarily  as 
training  aides  for  illustrations  and  graphics  (e.g.,  diagrams,  maps,  illustration  of  equipment,  route 
overlays). 
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Table  13 

Number  of  Courses  Using  ALM  Instructional  Techniques 


Instructional  Technique 

Number  (%)  of  Courses  Using  Technique 

Reduction  in  PowerPoint  and  instructor  lectures 

10  (100%) 

Facilitation 

10  (100%) 

Learner-centric  approach 

10  (100%) 

Specific  assessment  techniques 

10  (100%) 

Group  work 

9  (90%) 

Peer-to-peer  learning 

9  (90%) 

Hands-on  practical  exercises 

8  (80%) 

Homework  /  read  aheads 

8  (80%) 

Tailored  training 

6  (60%) 

Capstone  exercises 

5  (50%) 

Facilitation.  All  courses  stressed  facilitation,  although,  as  discussed  previously  under 
the  section  on  Instructor  Knowledge  and  Understanding  of  ALM,  this  was  not  necessarily  an 
easy  skill  for  instructors  to  acquire.  Another  difficulty  with  facilitation  was  when  a  portion  of 
the  students  in  the  courses  had  limited  background/  knowledge  regarding  the  content,  making  it 
hard  for  instructors  to  stimulate  student  discussions  and  participation.  Facilitation  also  was 
difficult  with  a  1:20  or  greater  instructor  to  student  ratio;  a  1:14  ratio  was  viewed  as  better. 

Techniques  to  increase  facilitation  were  cited.  One  technique  was  to  require  students  to 
provide  more  than  one-word  answers  and  to  ask  questions  of  the  student  who  did  not  contribute 
to  the  conversation  or  take  them  aside  for  a  one-on-one  ( Sustain ) 

Learner-centric.  Learner-centric  techniques  took  different  forms  depending  on  the 
content  and  student  populations.  However,  a  learner-centric  approach  was  typically  associated 
with  problem-solving  experiences,  having  students  be  responsible  for  their  own  learning,  and  in 
some  cases  role-playing  of  duty  positions.  It  also  meant  less  PowerPoint  and  sharing  of 
experiences.  For  some  of  the  more  advanced  courses,  the  learner-centric  approach  involved 
having  students  lead  the  exercises. 

Assessment.  Three  forms  of  assessment  were  used:  pre-tests,  subject  matter  tests,  and 

rubrics. 


Pre-tests.  The  need  for  pre-tests  was  acknowledged  but  typically  did  not  meet  the  need. 
Most  individuals  acknowledged  a  need  for  a  pre-test  to  help  them  identify  student  strengths  and 
weaknesses.  However,  pre-tests  did  not  always  function  appropriately;  they  did  not  necessarily 
identify  individuals  who  needed  assistance;  the  coverage  of  the  pre-test  was  too  narrow  and  did 
not  focus  on  current  lesson  plans.  These  comments  reflect  findings  in  prior  ARI  work  on  pre¬ 
tests  (Schaefer  &  Dyer,  2014;  Stallings,  Dyer,  Wampler  &  Cobb,  2014),  that  a  good  pre-test 
must  assess  relevant  prior  knowledge.  A  unique  use  of  a  pre-test  in  one  course  was  to  motivate 
students  by  showing  them  what  they  did  not  know. 
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Subject  matter  tests.  Traditional  subject  matter  tests  were  retained  for  grading  students 
but  were  replaced  by  more  challenging  tests  in  the  more  advanced  courses.  Most  instructors 
indicated  they  used  exams,  tests,  and/or  performance  tests  during  courses  to  assess  students  to 
grade  students.  In  some  cases  where  the  focus  was  on  hands-on  testing  or  group  testing, 
instructors  said  they  also  needed  written  tests.  Because  the  focus  in  more  advanced  courses  was 
on  the  ability  to  think  or  solve  problems,  the  tests  were  often  changed  (e.g.,  no  more  open-book, 
fewer  multiple-choice  or  true-false,  less  memorization,  not  all  tested  material  presented  in  class). 
Scenarios  could  also  be  inserted  in  test  items  which  required  students  to  solve  problems.  Such 
test  revisions  resulted  in  negative  responses  by  students  who  did  not  assume  responsibility  for 
their  own  learning,  as  was  required  in  some  course  revisions. 

Rubrics.  There  were  mixed  reactions  to  rubrics,  with  the  greatest  concern  being  the  time 
required  to  apply  them  as  intended.  Typically  those  interviewed  indicated  problems  with  rubrics 
to  include  insufficient  time  to  use  them  (e.g.,  no  time  for  both  a  pre-test  and  rubrics),  difficult  to 
turn  into  a  grade,  not  applied  appropriately,  and  difficulties  in  assessing  young  Soldiers  with 
rubrics.  Some  instructors  successfully  applied  rubrics  to  21st  Century  competencies  or  in 
counseling  sessions.  And  some  instructors  thought  their  rubrics  would  be  a  good  model  for 
others.  Also  rubrics  were  perceived  as  easier  to  apply  in  the  field  than  in  the  classroom. 

Group  work.  With  the  shift  from  a  lecture  mode  of  instruction,  the  most  common 
alternative  was  to  conduct  work  in  small  groups.  For  instance,  when  major  items  of  tactical 
equipment  were  used,  then  group  work  was  typical.  Groups  were  also  used  to  pair  more 
experienced  students  with  weaker  students  which  facilitated  peer-to-peer  learning  and  was  also 
viewed  by  most  as  a  form  of  tailored  training.  If  the  instructor  to  student  ratio  was  1  to  30, 
instructors  found  it  more  difficult  to  break  into  groups,  but  could  sometimes  pair  the  strong  with 
the  weak.  Often  group  work  was  followed  by  a  briefing  to  the  class  as  a  whole,  with  instructors 
insuring  that  more  than  one  person  briefed.  One  instructor  estimated  that  in  the  legacy  course 
there  was  little  to  no  group  work,  whereas  with  the  revised  course  it  was  60  to  70%. 

Peer-to-peer  learning.  Peer-to-peer  learning  and  peer  interaction  were  relatively 
common  because  of  the  shift  to  small  group  work.  Instructors  could  deliberately  pair  stronger 
with  weaker  students,  but  in  other  cases,  the  pairing  or  interaction  with  peers  would  happen  as  a 
matter  of  course.  Sometimes  “high  speed”  groups  who  finished  early  would  help  other  groups. 
In  general,  the  instructors  indicated  that  students  liked  being  in  groups  where  they  could  talk  to 
and  learn  from  each  other.  Instructors  found  that  students  explained  concepts  in  a  different  way 
to  get  points  across.  However,  instructors  had  to  monitor  carefully  to  insure  explanations  and 
concepts  were  correct. 

Hands-on  practical  exercises.  Hands-on  practical  exercises  were  common  when 
equipment  was  involved  in  basic  courses,  with  instructors  monitoring  students  to  assist  in 
discussion  of  issues  and  problems.  For  example,  instructors  would  try  to  get  students  to  think 
about  the  problem,  particularly  when  they  were  going  in  the  wrong  direction.  Instructors  would 
pose  questions  about  student  decisions,  not  just  tell  them  they  were  wrong.  The  hands-on 
exercises  were  viewed  as  one  way  of  developing  the  technical  competence  needed,  as  instructors 
believed  that  students  retain  less  when  they  are  “talked  at.”  Hands-on  exercises  also  led  to 
repetition,  which  was  viewed  as  increasing  retention  of  information.  Decision-making  and 
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planning  exercises  also  fell  in  this  instructional  category.  In  these  courses,  the  hands-on 
exercises  were  graded. 

Homework/read-aheads.  With  the  ALM  emphasis  on  decision-making  and  problem 
solving  and  challenging  students,  many  of  the  courses  shifted  the  burden  of  learning  from  the 
instructor  to  the  student  with  homework  or  readaheads. 

When  assigning  homework  or  read-aheads  for  the  next  class,  students  were  to  come 
prepared  to  the  class  with  this  newly-acquired  knowledge.  To  illustrate  the  change  in  emphasis 
and  why  homework/read-aheads  were  needed,  in  one  revised  course  the  instructors  said  only 
25%  of  what  the  student  was  tested  on  was  in  the  lesson  plans.  The  rest  of  the  information  had  to 
be  found  outside  of  the  course,  whereas  in  the  old  program  of  instruction,  the  instructors  covered 
everything  in  the  tests. 

There  were  both  positive  and  negative  effects  of  this  change.  For  a  basic  course, 
instructors  found  that  not  all  students  had  time  to  get  to  the  homework  assignment.  For  more 
advanced  courses,  instructors  also  found  that  not  all  students  did  their  homework  or  read-aheads, 
which  was  evident  in  their  inability  to  participate  in  class  or  in  a  high  failure  rate  on  exams. 
Some  students  did  not  adapt  to  this  shift  in  responsibility  and  reacted  negatively.  Other  students 
reacted  very  positively,  rose  to  meet  the  challenge,  and  enjoyed  this  form  of  training. 

Tailored  training.  When  classes  were  composed  of  students  with  diverse  backgrounds, 
instructors  were  challenged  on  how  to  adapt  to  this  diversity.  Some  instructors  overcame  this 
challenge  by  tailoring  the  training  to  the  varying  levels  of  student  expertise.  The  most  extensive 
comments  were  provided  by  individuals  representing  the  Sustain  stage. 

Tailored  training  was  addressed  in  different  ways,  although  it  appeared  tailored  training 
techniques  were  not  deliberately  incorporated  into  the  courses  but  typically  left  to  the  discretion 
of  the  instructor.  One  common  means  was  to  pair  strong  with  weak  students  to  enable  students 
to  teach  students.  This  approach  was  consistent  with  the  findings  by  Dyer,  Wampler  and 
Blankenbeckler  (2011).  When  there  were  sufficient  instructors,  instructors  took  struggling 
students  aside  and  worked  with  them.  A  limited  number  of  instructors  indicated  they  tried  to 
challenge  those  who  were  more  capable  or  who  wanted  to  achieve  more  by  giving  them 
additional  exercises.  In  other  cases,  instructors  indicated  that  they  couldn’t  tailor  because  of 
limited  time  and/or  that  challenging  the  more  capable  individuals  was  difficult  because  of 
constraints  on  the  number  of  instructors. 

Capstone  exercises.  Capstone  exercises  existed  in  half  the  courses.  They  varied  in 
scope  and  were  not  new  to  the  course,  e.g.,  a  team-based  exercise,  a  planning  exercise,  a 
field  training  exercise,  and  an  “engagement  activity.”  In  one  course  a  capstone  exercise  was 
added,  requiring  development  of  a  plan,  briefing  the  plan,  and  executing  it. 
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Relationship  among  training  techniques.  The  picture  which  emerged  from  these 
profiles,  as  illustrated  in  Figure  5,  was  that  reliance  on  PowerPoint  (a  “sage  on  a  stage”)  was 
replaced  by  efforts  to  facilitate  learning  and  thinking  via  the  instructor  and/or  through  small 
groups,  peer-to-peer  interactions,  and  hands-on  practical  exercises.  More  responsibility  was 
placed  on  the  students  with  home-work  and  read-ahead  materials  as  well  as  a  change  in  test 
procedures.  PowerPoint  slides  were  greatly  reduced  but  not  necessarily  eliminated.  Tailored 
training  was  of  concern  when  the  student  population  had  diverse  backgrounds.  Of  note,  is  that 
new  technologies  or  simulations  were  not  introduced;  existing  technologies  were  used. 
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Figure  5.  Relationship  among  training  techniques  and  approaches. 

Soldier  Competencies 

The  results  for  the  21st  Century  Soldier  competencies  stressed  in  the  courses  were  based 
primarily  on  the  interviews.  For  courses  in  the  Plan/Dev  stage,  the  competencies  were  based  on 
the  plan  and  lesson  development  at  that  point.  There  were  three  divisions  to  the  competency 
frequencies.  Two  (character  and  accountability,  and  tactical  and  technical)  were  stressed  in  all 
courses,  four  (adaptability  and  initiative,  critical  thinking  and  problem-solving,  communication 
and  engagement,  teamwork  and  collaboration)  were  stressed  in  most  courses,  and  three  (life-long 
learner,  comprehensive  fitness,  and  cultural  and  JEM)  were  used  in  two  or  fewer  courses.  The 
most  common  competencies  that  were  stressed  are  discussed  below. 
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Character  and  accountability.  Accountability  was  stressed  by  placing  more 
responsibility  on  students  for  own  learning  through  homework,  read-aheads  and/or  nature  of 
course  test.  Instructors  often  instituted  consequences  when  preparatory  work  was  not  done. 
Instructors  reported  behavioral  changes  in  students  when  they  did  not  perform  well  on  tests.  For 
some  advanced  courses,  accountability  demands  were  viewed  as  a  cultural  change.  Some 
instructors  felt  that  placing  students  in  a  leadership  was  also  a  form  of  accountability. 

Tactical  and  technical.  The  revised  course  gave  students  the  opportunity  to  practice  job 
skills  and  to  be  tested  on  them.  Primary  course  purpose  was  tactical  and  technical 
competency/expertise. 

Adaptability  and  initiative.  In  the  Sustain  and  Plan/Dev  stages,  changing  training 
scenarios  and  conditions  or  responsibilities  in  training  situations  were  viewed  as  techniques 
which  required  students  to  adapt. 

Critical  Thinking  and  problem  solving.  For  the  Sustain  and  Execute  stages,  the  revised 
course  was  viewed  as  providing  opportunities  to  solve  problems  through  classroom  and/or  field 
exercises.  In  the  more  advanced  courses  in  the  Sustain  stage,  research  papers  also  facilitated  this 
competency. 

Teamwork  and  collaboration.  For  the  Sustain  and  Plan/Dev  stages,  increased  emphasis 
on  group  activities  was  viewed  to  foster  teamwork  and  collaboration.  For  the  Execute  stage, 
field  training  exercises  and  equipment  set-up  procedures  were  viewed  to  foster  this  competency. 

Communication  and  engagement.  The  consensus  was  that  student  briefings  fostered 
communication  skills. 

Lessons  Learned  From  the  Follow-on  of  Course  Graduates 

In  the  follow-on  effort,  30  graduates  of  a  revised  course,  23  graduates  of  a  legacy  course, 
and  25  leaders  were  interviewed.  The  main  findings  are  reported  below  and  provide  good 
lessons  learned  for  enhancing  learning  outcomes  from  a  training  context  to  ‘on-the-job 
performance’  (see  Transfer  of  Training  Lessons  on  page  39). 

Several  challenges  were  encountered  in  linking  changes  in  the  course  mode  of  instruction 
to  duty  position  performance.  First,  the  eventual  unit  position  can  differ  from  the  focus  of  the 
course.  Consequently,  some  graduates  stated  that  the  course  did  not  prepare  them  for  certain 
duty  position  requirements.  Second,  unit  leaders  often  had  difficulties  in  determining  whether 
the  graduates’  performance  in  the  unit  resulted  from  the  training  or  from  the  students’  prior 
experiences  and  abilities. 

Comments  were  made  by  course  graduates  on  ALM  techniques.  The  students’ 
backgrounds  and  knowledge  in  this  course  were  quite  diverse.  Graduates  indicated  a  desire  for 
the  course  instructors  to  leverage  this  experience  and  knowledge  rather  than  treating  everyone 
similarly  (i.e.,  wanted  more  tailored  training).  On  the  other  hand,  the  graduates  did  react 
positively  to  being  required  to  brief,  teach  classes  to  others,  and  mentor  others  with  less 
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experience  and  knowledge.  They  also  valued  the  hands-on  practical  exercises  and  wanted  more 
of  these.  Graduates  with  prior  relevant  knowledge  and  experience  benefitted  more  from  the 
course  as  they  were  able  to  understand  how  the  course  related  to  their  future  duty  positions. 

Conclusions  and  Recommendations 

This  section  provides  the  main  findings  from  the  current  research  as  well  as 
recommended  strategies  for  implanting  large-scale  curriculum  changes  in  the  Army  School 
System.  See  Appendix  E  for  supplemental  information  on  the  effects  of  some  of  the  facilitating 
and  inhibiting  factors. 

A  major  conclusion  from  the  research  is  that  the  timeframe  to  implement  substantial 
curriculum  is  1.5  to  2  years  with  sufficient  time  to  ensure  the  changes  are  being  implemented  as 
intended.  The  process  is  complex  as  it  involves  different  organizations,  must  be  sufficiently 
robust  to  withstand  personnel  turbulence,  and  requires  that  all  personnel  involved  are  adequately 
trained  on  the  new  requirements  and  new  techniques. 

The  change  process  followed  prior  research  findings  on  the  diffusion  of  innovations 
(Rogers,  2003),  with  the  facilitating  and  inhibiting  factors  also  reflecting  what  has  been 
documented  in  this  body  of  research.  Factors  that  facilitated  the  process  were  when: 

•  Personnel  viewed  the  ALM  as  relevant  and  viable  for  the  course  and  benefiting  students, 

•  There  was  initial  agreement  among  key  players  on  the  revisions, 

•  There  was  a  minimal  paradigm  shift  in  instructional  approaches, 

•  Both  training  developers  and  instructors  were  trained  well  on  ALM, 

•  Pilots  of  the  course  were  conducted, 

•  Training  developers  and  instructors  coordinated  regarding  both  the  changes  themselves 
and  implementation  of  the  changes, 

•  Reduced  use  of  PowerPoint  “forced”  a  change  in  instructional  approaches  to  facilitation 
and  small  group  work;  there  was  a  champion  who  insured  continual  progress  in  all  stages 
of  innovation,  and. 

•  The  instructors  and  training  developers  worked  very  hard  at  implementing  ALM  concepts 
and  contributed  to  the  successes  that  occurred. 

Factors  that  inhibited  the  process,  which  primarily  impacted  training  developer  workload  and 
lengthened  the  timeline,  were  when: 

•  Training  developer  workload  was  high, 

•  Turbulence  occurred  in  key  positions:  course  manager,  training  developer,  instructor, 

•  There  were  shortages  of  instructors, 

•  Higher-level  factors  required  changes  in  addition  to  ALM,  and 

•  There  was  insufficient  guidance  on  how  to  develop  lesson  plans  to  facilitate  critical 

thinking  and  problem-solving. 
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Factors  that  made  the  ALM  process  challenging  for  instructors  were  when: 


•  Substantial  individual  differences  in  prior  knowledge  and/or  experience  occurred  within 
the  student  population, 

•  Greater  subject  matter  knowledge  was  required  to  be  a  good  facilitator;  instructors 
needed  more  training  on  facilitation  techniques. 

•  Instructors  often  had  difficulties  in  making  major  behavioral  changes  which  were 
inconsistent  with  how  they  had  been  taught  or  how  they  had  been  teaching,  and 

•  Not  all  students  reacted  positively  to  being  more  accountable  for  their  own  learning. 

Another  critical  factor  is  to  assess  the  impact  of  the  changes.  Decision-makers  want  to 
know  if  the  desired  changes  were  implemented  by  instructors.  If  the  answer  is  affirmative,  then 
the  next  question  becomes  did  these  changes  impact  students.  What  was  the  return  on 
investment?  Did  students  learn  more,  did  they  retain  information  longer,  could  they  apply  the 
information  more  effectively,  did  they  have  a  more  in-depth  understanding  of  the  material,  etc.? 
However,  the  typical  focus  found  in  the  research  was  on  making  the  curriculum  changes,  not  on 
planning  for  or  executing  a  systematic  assessment  of  student  performance.  Although  students 
may  be  surveyed  regarding  their  reactions  to  changes,  this  is  not  equivalent  to  obtaining  data  on 
student  performance  or  proficiency.  Whenever  possible,  the  evaluation  plan  to  assess  the  effects 
of  the  major  curriculum  changes  should  include  formed,  objective  measures  of  student 
performance.  These  assessments  should  occur  before  and  after  the  changes  have  been  made  to 
address  decision-makers’  questions  on  the  value  of  such  changes  (not  while  the  changes  are  in 
progress). 

Sustainment  of  Curriculum  Changes 

The  following  four  factors  were  identified  from  the  research  findings  that  affect  the 
degree  to  which  the  curriculum  changes  will  be  sustained  over  time. 

Execute  pilot  efforts.  Curriculum  developers  should  plan  to  execute  the  course  changes 
via  pilots  (across  several  iterations  of  the  course).  The  findings  indicated  that  the  courses  that 
adopted  this  method  accelerated  the  implementation  of  ALM  and  indicated  that  the  course  was  in 
a  steady  state  (only  minor  changes  were  expected  in  the  near  future).  This  was  due  in  part  to  the 
faster  feedback  cycle  between  training  developers  and  instructors  such  that  the  training 
developers  learned  of  any  revisions  that  needed  to  be  made  to  the  lesson  plans  at  a  much  faster 
rate. 


Communicate  the  tenets  of  the  proposed  curriculum  change.  Organizational 
procedures  for  communicating  the  tenets  of  ALM  as  well  as  how  the  organization  has  adopted 
these  tenets  to  new  personnel  are  critical  to  sustain  the  curriculum  changes. 

Train  instructors  on  how  to  implement  the  changes.  A  key  factor  underlying  the 
majority  of  the  research  findings  and  in  sustaining  curriculum  changes  is  how  well  instructors  are 
trained  to  implement  the  types  of  techniques  required  by  the  ALM.  As  new  instructors  are 
assigned  to  courses,  these  instructors  need  to  know  the  overall  purpose  of  the  ALM  and  the 
specific  techniques,  methods,  and  processes  for  implementing,  sustaining,  and  furthering  the 
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ALM  tenets  within  Army  courses.  Leaders  should  support  sending  instructors  to  follow-on 
courses  to  the  Army  Basic  Instructor  Course  (ABIC)  and  the  Small  Group  Instructor  Training 
Course  (SGITC)  to  further  develop  tailored  training  and  facilitation  techniques.  If  instructors 
have  additional  opportunities  to  learn  and  practice  facilitation  techniques,  they  will  be  more  apt 
to  use  these  techniques  in  the  classroom  and  share  these  new  techniques  with  other  instructors. 

Obtain  feedback  on  success  of  course  changes.  Participants  indicated  a  need  to  obtain 
feedback  from  course  graduates  when  they  are  in  their  gaining  units.  Participants  indicated  that 
these  data  would  be  helpful  in  determining  the  success  of  the  course  changes  and  if  further 
revisions  are  needed. 

Transfer  of  Training  Lessons  Learned  from  Follow-On  Research 

To  maximize  transfer  of  learning  outcomes,  tailored  training  approaches  need  to  be 
implemented  and  sustained  in  the  courses.  To  better  retain  content,  students  and  instructors  who 
have  lower  levels  of  background  knowledge  and  experience  need  more : 

•  Instructor  coaching/mentoring  and  feedback 

•  Opportunities  to  practice  tasks  and  master  content  (e.g.,  additional  exercises,  teaching 
experiences,  role  playing) 

•  Peer-to-peer  learning  opportunities  to  benefit  from  experienced  students/instructors 
including  observing  task  demonstrations 

Finally,  to  maximize  application  of  ALM  methods  by  course  graduates,  gaining  units 
need  to  implement  processes  that  support  the  tenets  of  ALM.  Although  the  primary  tenets  of 
ALM  are  focused  on  increasing  student  learning,  sub-goals  of  the  Army  Learning  Concept  2015 
are  to  provide  instructors  and  trainers  with  techniques  that  they  can  use  both  in  institutional  and 
operational  units.  Unit  leaders  who  create  environments  for  Soldiers  to  conduct  training  based 
on  their  knowledge  and  experience  executing  the  ALM  can  maximize  and  enhance  the  widely 
utilized  train-the-trainer  method. 

Roadmap  for  Success 

The  following  recommendations  are  made  for  implementing  major  curriculum  changes  or 
innovations  within  Army  institutional  courses  (see  the  flow  chart  in  Figure  6  on  the  next  two 
pages).  The  recommendations  stress  the  importance  of  a  good  plan  to  include  performance 
assessment  and  involvement  by  all  key  players. 
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Decision-making  and  Planning  Phases 


•Obtain  agreement  and  input  from  all  key  players  on  the  approach 

•Identify  individual/unit  responsible  for  overseeing  implementation  of  innovation  plans 


•  Conduct  required  training  analysis  to  support  the  approach 

•  Implement  training  plan  for  key  players 

•  Estimate  degree  of  change  to  determine  resources  (personnel,  time)  required  during  later  phases 


•  Plan  for  a  phased  implementation  approach  (pilot  the  course  or  course  lessons)  with  more  than 
one  implementation  and  with  systematic  observation  of  training 
•Plan  for  baseline  assessment  of  student  performance 


•  Determine  if  there  are  other  higher-level  changes  to  be  made  simultaneously  and  must  be 
considered  in  workloads  and  timelines 


Development  Phase 


•Establish  a  policy  for  interaction  between  training  developers  and  instructors 
•Review  draft  lesson  plans  to  ensure  vision  and  intent  of  changes  are  being  made 


•Determine  if  there  are  unanticipated  effects  when  development  is  underway  (e.g.,  shortage  or 
turbulence  of  personnel)  and  make  necessary  adjustments 
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Execution  Phase 


•Ensure  there  is  an  observer  monitoring  the  execution  of  changes  who  provides  instructor 
feedback  to  training  developers 


•Obtain  feedback  from  students,  when  possible 
•Determine  whether  feedback  is  used  to  further  revise  lesson  plans,  and  if  necessary,  to  modify  the 
original  innovation 


End 

right 


Resources 

Again 

\/ 

Student 

Outcomes 


> 

•  Sustainment  Phase 

•Ensure  resources  are  adequate  to  sustain  instructional  changes  (e.g.,  instructor  training 
to  reduce  negative  impacts  of  instructor  turbulence) 

> 

•Assess  student  reaction  to  course 

•Assess  student  performance  and  compare  with  baseline  performance 


Figure  6.  Roadmap  for  Success:  Recommendations  for  the  implementation  of  curriculum 
innovations. 
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Appendix  A 

21st  Century  Soldier  Competencies 


Character  and  accountability 

a.  Soldiers  and  leaders  demonstrate  Army  values,  the  Soldier’s  Creed,  and  Warrior  Ethos  through 
action  while  also  developing  character  and  accountability  in  subordinates.  They  accept 
obligations  of  service  before  self  and  for  assigned  tasks,  missions,  their  subordinates,  and 
themselves  while  building  confidence  in  leaders,  peers,  and  subordinates  that  they  can  be 
counted  upon  to  accomplish  goals.  Soldier  and  leader  actions  are  guided  by  the  Army  Ethic, 
which  consists  of  the  shared  values,  beliefs,  ideals,  and  principles  held  by  the  Army  Profession 
of  Arms  and  embedded  in  its  culture  that  are  taught  to,  internalized  by,  and  practiced  by  all 
Soldiers  in  full-spectrum  operations  as  well  as  peacetime. 

b.  Adhering  to  and  internalizing  the  Army  Ethic  develops  strong  character,  ethical  reasoning  and 
decision-making,  empathy  for  others,  and  the  self-discipline  to  always  do  what  is  right  for  fellow 
Soldiers,  the  Army,  and  the  Nation.  Character  enables  the  Soldier  to  operate  in  a  complex  and 
uncertain  environment  with  the  understanding  that  the  Soldier  is  individually  accountable  for  not 
only  what  is  done,  but  also  for  what  might  not  be  done.  The  pride,  esprit,  and  ethos  required  of 
Soldiers  as  members  of  the  Profession  of  Arms  may  require  them  to  sacrifice  themselves 
willingly  to  preserve  the  Nation,  accomplish  the  mission,  or  protect  the  lives  of  fellow  Soldiers. 
Qualities  of  character  and  ethical  behavior  will  be  stressed  at  every  level. 

Comprehensive  fitness 

Soldiers  and  leaders  develop  and  maintain  individual,  as  well  as  that  of  their  subordinates, 
physical,  emotional,  social.  Family,  and  spiritual  fitness.  They  display  physical,  mental,  and 
emotional  persistence,  quickly  recover  from  difficult  situations,  and  exemplify  the  resilience 
necessary  to  fight  and  win  in  any  operational  situation. 

Adaptability  and  initiative 

a.  Soldiers  and  leaders  are  comfortable  operating  in  unexpected  situations  throughout  the  world. 
They  scan  the  environment,  identify  unique  or  unexpected  conditions,  and  adjust  to  handle  the 
situation  effectively. 

b.  Soldiers  and  leaders  recognize  when  standard  procedures  are  not  an  effective  solution  to  a 
situation  and  use  innovation  to  develop  new  procedures,  devices,  and  others,  that  are  necessary 
to  handle  the  situation.  Mental  agility  and  a  global  mindset  allow  them  to  anticipate  changes  in 
the  operational  environment,  adapt  to  the  changes,  and  anticipate  the  second  and  third  order 
effects  of  their  actions  and  decisions. 

c.  Soldiers  and  leaders  take  appropriate  action  and  calculated  risks  in  the  absence  of  orders  or  in 
situations  that  require  modifying  orders  to  achieve  the  commander’s  intent  while  also  developing 
initiative  and  risk  taking  in  subordinates.  They  anticipate  changes  in  the  operational  environment 
assess  the  situation  and  use  sound  judgment  to  decide  when  and  how  to  act.  Self-awareness 
allows  Soldiers  and  leaders  to  monitor  and  adjust  their  actions  and  those  of  their  teams  to 
constantly  assess  performance  and  seek  improvement. 
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Lifelong  learner  (includes  digital  literacy) 

a.  Soldiers  and  leaders  continually  assess  themselves,  identify  what  they  need  to  learn  and  use 
skills  that  help  them  to  effectively  acquire  and  update  knowledge,  skills,  and  attitudes.  Soldiers 
and  leaders  value  and  integrate  all  forms  of  learning  (formal,  informal)  on  a  daily  basis  to  seek 
improvement  of  themselves  and  their  organizations  continuously. 

b.  Soldiers  and  Leaders  access,  evaluate,  and  use  information  from  a  variety  of  sources  and 
leverage  technology  (hardware  and  software)  to  improve  their  effectiveness  and  that  of  their 
teams  while  executing  the  Army’s  missions.  Digital  literacy  skills  are  developed  at  initial  entry 
and  increase  progressively  at  each  career  level. 

Teamwork  and  collaboration 

Soldiers  and  leaders  create  high-performing  formal  and  informal  groups  by  leading,  motivating, 
and  influencing  individuals  and  partners  to  work  toward  common  goals  effectively.  They  are 
effective  team  members,  understand  team  dynamics,  and  take  appropriate  action  to  foster  trust, 
cohesion,  communication,  cooperation,  effectiveness,  and  dependability  within  the  team.  Leaders 
build  teams,  seek  multiple  perspectives,  alternative  viewpoints,  and  manage  team  conflict. 

Communication  and  engagement  (oral,  written,  and  negotiation) 

a.  Soldiers  and  leaders  express  themselves  clearly  and  succinctly  in  oral,  written,  and  digital 
communications.  They  use  interpersonal  tact,  influence,  and  communication  to  build  effective 
working  relationships  and  social  networks  that  facilitate  knowledge  acquisition  and  provide 
feedback  necessary  for  continuous  improvement. 

b.  Soldiers  and  leaders  inform  and  educate  U.S.,  allied,  and  other  relevant  publics  and  actors  to 
gain  and  maintain  trust,  confidence,  and  support.  Engagement  is  characterized  by  a 
comprehensive  commitment  to  transparency,  accountability,  and  credibility,  and  is  an  imperative 
of  21st  century  operations. 

Critical  thinking  and  problem  solving 

Soldiers  and  leaders  analyze  and  evaluate  thinking,  with  a  view  to  improving  it.  They  solve 
complex  problems  by  using  experiences,  training,  education,  critical  questioning,  convergent, 
critical,  and  creative  thinking,  and  collaboration  to  develop  solutions.  Throughout  their  careers, 
Soldiers  and  leaders  continue  to  analyze  information  and  hone  thinking  skills  while  handling 
problems  of  increasing  complexity.  Select  leaders  also  develop  strategic  thinking  skills  necessary 
for  assignments  at  the  national  level. 

Cultural  and  joint,  interagency,  intergovernmental,  and  multinational  competence 

Soldiers  and  leaders  use  cultural  fundamentals,  self-awareness  skills,  and  regional  competence  to 
act  effectively  in  any  situation.  They  use  communication,  including  foreign  language,  influence, 
and  relational  skills  to  work  effectively  in  varied  cultural  and  joint,  interagency, 
intergovernmental,  and  multinational  contexts.  Soldiers  and  leaders  consider  and  are  sensitive  to 
socially  transmitted  behavior  patterns  and  beliefs  of  individuals  from  other  communities  and/or 
countries  and  effectively  partner,  influence,  and  operate  in  complex  joint,  interagency, 
intergovernmental,  and  multinational  environments. 
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Tactical  and  technical  competence  (full  spectrum  capable) 

a.  Soldiers  and  leaders  employ  tactical  and  technical  skills  in  full-spectrum  operations  to 
accomplish  the  mission  and  support  the  commander’s  intent.  They  are  experts  on  weapons 
systems,  combined  arms  operations,  and  train  their  subordinates  to  be  technically  and  tactically 
competent.  At  lower  levels,  they  are  technical  experts  in  their  specialty  and  continue  to  develop 
their  technical  skills  and  those  in  their  subordinates.  As  leaders  grow,  they  increase  their 
understanding  and  application  of  mission  command,  operational  contexts,  systems,  and 
technology  while  operating  in  increasingly  complex  environments. 

b.  Soldiers  and  leaders  are  prepared  to  execute  offensive,  defensive,  stability,  and  civil  support 
missions  throughout  the  continuum  of  operations  and  transition  between  diverse  tasks  and 
operational  actions  as  complex  and  uncertain  operational  situations  are  developed  through  action. 
Leaders  anticipate  tactical,  operational,  and  strategic  transitions  and  use  mission  command  to 
apply  lethal  and  nonlethal  effects  to  achieve  the  commander's  intent. 


Source:  Department  of  the  Army.  (2011).  The  U.  S.  Army  learning  concept  for  2015  (TRADOC 
Pamphlet  525-8-2).  Ft.  Monroe,  VA:  Training  and  Doctrine  Command,  pages  41-43. 
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INTRODUCTION  TO  INTERVIEW  QUESTIONS 


In  2012,  TRADOC  G-3/5/7  initiated  the  Shot-in-the-Arm  (SITA)  Phase  I  Instructional  Design  Course  (IDC). 
CAC  LDE  was  the  lead  to  support  schools  in  developing  skill  sets  to  implement  the  Army  Learning  Model 
(called  ALM).  The  next  year,  in  2013,  G-3/5/7  initiated  SITA  Phase  II  Adaptive  Soldier  and  Leader 
Training  and  Education  (ASLTE)  Mobile  Training  Team  effort.  The  Asymmetric  Warfare  Group  (AWG) 
was  the  lead.  SITA  Phase  II  ASLTE  was  identified  as  a  CG,  TRADOC  priority  and  each  school  was  tasked  to 
revise  one  course  integrating  ALM  concepts  and  ASLTE  principles. 

TRADOC  G-3/5/7  requested  that  ARI  examine  a  sample  of  courses  to  identify  lessons  learned  regarding 
the  implementation  process  as  ALM  concepts  and  ASLTE  principles  were  integrated  into  revised  courses. 
The  _ [insert  name  of  course] _ was  one  of  the  courses  identified  by  TRADOC. 

We  are  interested  in  understanding  how  the  implementation  process  evolved/occurred  here  at  your 

school.  We  will  start  with  the  decision  to  select  _ [insert  name  of  course] for  TRADOC's  SITA  effort, 

continue  through  your  entire  planning  and  development  processes  and  end  with  your  plans  to  sustain 
the  course  revisions. 

We  are  interviewing  individuals  involved  with  the  implementation  of  ALM  concepts  in _ [insert  name 

of  course] _ .  This  includes  training  developers,  instructors,  course  managers,  staff  and  faculty,  quality 

assurance  individuals,  and  appropriate  leaders.  We  know  that  these  individuals  have  different  roles, 
and  the  interview  will  focus  on  your  specific  role.  First  we  will  ask  you  to  complete  some  background 
questions  to  gain  an  understanding  of  your  major  roles  and  responsibilities.  Follow-on  questions  will 
then  be  directed  toward  your  specific  role. 

We  greatly  appreciate  your  participation  in  this  effort.  It  will  allow  us  to  provide  TRADOC  G-3/5/7  an 
analysis  and  recommendations  on  lessons  learned  and  how  they  can  improve  the  implementation 
process  in  the  future. 


B-2 


BACKGROUND  INTERVIEW  QUESTIONS 


1.  Course  information 

Note,  ARI  may  be  able  to  obtain  this  course  information  from  the  Point  of  Contact  at  the 
School  prior  to  our  visit.  If  obtained,  the  course  information  questions  will  not  be  asked. 

a.  Class  size _ 

b.  Number  of  military  instructors _ civilian _ 

c.  Length  of  course _ 

d.  Frequency  of  course _ 

e.  Specific  Army  equipment  /  ranges  /  facilities  used _ 

f.  Student  Population _ 


Interviewer: 

•  Request  the  individuals  complete  the  background  questionnaire.  Indicate  that  you  will  be 
asking  some  follow-up  questions. 

•  Collect  all  the  completed  questionnaires  and  examine  the  responses  to  each  question. 


SITA 

2.  Did  you  attend  SITA  Ph  I  Instructional  Design  Course  (IDC)? _ Yes  _ No 

a.  If  yes: 

Was  SITA  I  held  prior  to  or  following  the  selection  of  the  revised  course?  Prior  After 
Was  SITA  I  held  prior  to  or  following  the  planning  for  the  revised  course?  Prior  After 

b.  What  was  the  timeframe  for  SITA  Ph  I? 

c.  What  are  the  main  concepts/  ideas/  techniques  you  remember  from  the  SITA  Ph  I? 
Probe  on  the  following 

i.  Learner  centric  training  (what  does  this  mean  to  you?) 

ii.  Soldier  competencies 

iii.  Developing  outcome  statements 

iv.  Blended  learning 

v.  Tailoring 

vi.  Training  technologies 

vii.  Measurement  of  competencies 

viii.  Developing  lesson  plans 

d.  What  was  the  content  and  method  of  the  student-focused  exercises  in  which  you 
participated  in  SITA  Ph  I? 
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3.  Did  you  attend  SITA  Phase  II  Adaptive  Soldier  and  Leader  Training  and  Education  (ASLTE) 

MTT  in  2013? _ Yes  _ No 

a.  If  yes: 

Was  SITA  II  held  prior  to  or  following  the  selection  of  the  revised  course?  Prior 
After 

Was  SITA  II  held  prior  to  or  following  the  planning  for  the  revised  course?  Prior 
After 

b.  What  was  the  timeframe  for  SITA  II? 

c.  What  are  the  main  concepts/  ideas/  techniques  you  remember  from  the  SITA  II? 
Probe  on  the  following 

i.  Learner  centric  training  (what  does  this  mean  to  you?) 

ii.  Soldier  competencies 

iii.  Developing  outcome  statements 

iv.  Blended  learning 

v.  Tailoring 

vi.  Training  technologies 

vii.  Measurement  of  competencies 

viii.  Developing  lesson  plans 

d.  What  was  the  content  and  method  of  the  student-focused  exercises  in  which  you 
participated? 

4.  Did  you  attend  a  SITA  Ph  II  Follow-on  MTT? _ Yes  _ No 

a.  What  was  the  timeframe  for  the  SITA  Ph  II  Follow-on  MTT? 

b.  What  are  the  main  concepts/  ideas/  techniques  you  remember  from  the  SITA  Follow-on 
MTTs? 

Probe  on  the  following 

(i)  Learner  centric  training  (what  does  this  mean  to  you?) 

(ii)  Soldier  competencies 

(iii)  Developing  outcome  statements 

(iv)  Blended  learning 

(v)  Tailoring 

(vi)  Training  technologies 

(vii)  Measurement  of  competencies 
(viii)  Developing  lesson  plans 

c.  What  was  the  content  and  method  of  the  student-focused  exercises  in  which  you 
participated? 

5.  If  they  did  not  attend  a  SITA  Ph  II  ASLTE  MTT  ask  the  following: 

a.  What  have  you  been  told  about  SITA  Ph  II  ASLTE  and  its  intent? 
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Roles  and  Responsibilities 


6.  Before  we  go  on,  I  want  to  make  sure  I  understand  your  role  in  revising  the  course.  Your 
responses  in  the  questionnaire  indicated  the  following.  [Interviewer  -  check  response  and 
verify  with  participants.] 


Involved 

Aware 

Not  Involved 

Not  involved  but  should 

have  been 

Stage  1.  Decision-making 
(Course  selection) 

Stage  2.  Planning  & 
Identifying  Solutions 
(Analysis  and  Design) 

Stage  3.  Development 
(Production  of  materials, 
TDC  input) 

Stage  4.  Implementation 
(Course  execution/pilot 
course  in  the  classroom) 

Stage  5.  Sustainment 

7.  Let  me  verify  the  times  you  specified  for  each  stage  as  well. 


Stage  1.  Decision-making  (course  selection) 

Date  of  decision: 

Stage  2.  Planning  &  Identifying  Solutions 

From  to 

Stage  3.  Development 

Materials:  From  to 

Instructors  prepared  From  to 

Stage  4.  Implementation  (course  execution) 

First  iteration:  from  to 

Stage  5.  Sustainment 

Interviewer:  Review  the  arrows  the  participants  drew  on  the  organizational  chart  to  make  sure  your 
understanding  of  them  is  correct. 
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Before  we  progress  to  your  specific  role  and  responsibilities,  we  have  two  more  questions. 

8.  In  many  cases  when  major  changes  are  made  within  an  organization,  there  is  a  “champion" 
for  the  change.  A  champion  is  an  individual  who  takes  the  lead  in  conceptualizing  the 
changes  needed,  monitors/reviews  the  progress  being  made,  insures  the  right  people  are 
involved  at  the  right  times,  comes  up  with  solutions  to  unexpected  problems,  someone  who 
is  enthusiastic  about  the  changes,  is  determined  to  have  success,  etc.. 

a.  Was  (is)  there  such  an  individual  within  your  school  or  CoE  who  could  be  considered  a 
champion? 

Yes  No 

If  yes,  duty  position _ 

9.  From  your  perspective,  what  are  the  success  stories  associated  with  revisions  to  this  course? 
(probe,  increased  student  proficiency,  special  exercises,  something  that  solved  a  major 
training  gap,  something  that  you  think  should  be  repeated,  the  focus  on  "outcomes,"  tools 
that  made  the  process  more  efficient,  increased  student  engagement,  increase  in  positive 
response  from  students  ...). 

We  will  now  ask  you  about  your  specific  responsibilities  in  more  depth. 
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BACKGROUND  PARTICIPANT  QUESTIONNAIRE 


Background  Questions 

1.  Please  mark  your  duty  position: 

_ Instructional  designer 

_ Training  developer 

_ I  nstructor/Writer 

_ Instructor 

_ Course  manager 

_ Middle  manager 

_ Staff  and  faculty 

_ Quality  assurance  staff 

_ Other  (please  write  other  position) _ 

2.  How  long  have  you  held  this  position?  _ 

3.  Have  you  previously  held  any  other  positions  in  the  list  presented  above? 

_ Yes  _ No 

If  yes,  which  ones? _ 

4.  If  military.  What  is  your  rank? _ 

What  is  your  branch/MOS? _ 

5.  Please  review  your  organizational  chart  (See  the  last  page  of  this  questionnaire).  Is  this  the 
present  structure  of  your  organization? 

_ Yes  _ No 

6.  To  the  best  of  your  knowledge,  are  the  identified  units/agencies  on  this  chart  the  ones 
involved  in  the  ALM  course  changes? 

_ Yes  _ No 

7.  Did  you  attend  a  SITA  Ph  I  IDC  Train  the  Trainer  or  local  version  in  2012? 

_ Yes  _ No  _ Which? 

8.  Did  you  attend  a  SITA  Ph  II  ASLTEMTT  in  2013?  _ Yes  _ No 

9.  Did  you  (are  you)  attend  a  SITA  Ph  II  ASLTE  MTT  Follow-on  being  conducted  in  2014? 

Yes  No 
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The  next  items  address  your  general  role  in  the  five  stages  which  are  associated  with  making  changes 
within  organizations.  We  want  to  know  if  you  were  involved  in  each  stage  and  the  status  of  that 
stage. 

Stage  1.  Decision-Making  (Course  Selection) 

1.  Were  you  involved  in  making  the  decision  regarding  which  course  to  include  in  SITA  2? 

_ Yes  _ No 

a.  If  /Vo,  were  you  aware  of  the  fact  that  decisions  were  being  made? 

_ Yes  _ No 

b.  If  Not  involved,  do  you  think  you  should  have  been  involved? 

_ Yes  _ No 

(If  not  involved  or  not  aware,  go  to  questions  on  Stage  2) 

2.  Please  indicate  on  the  scale  below  the  best  description  of  this  decision-making  process. 


CHANGE  ACTIVITY 

Circle  the  cell  that  best  describes  the  decision-making  stage 

Identifying  the  course  to 
include  in  response  to  the 
SITA  Phase  II  TRADOC 
Tasking  Order 

■4 

Candidate  courses 
hard  to  identify; 

Decision  made  after 

much  discussion  and 

debate 

Several  courses 
immediately 
considered;  Pros  and 
cons  weighed  before 
final  decision 

Candidate  course(s) 
easily  identified. 
Course  decision- 

made  with  little 

debate 

3.  When  was  this  course  decision  made?  (Month/year) 
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Stage  2:  Planning  and  Identifying  Solutions 


4.  Were  you  involved  in  determining/discussing  the  types  of  changes  needed,  what  was 
feasible,  and/or  possible  solutions  for  revising  this  course? 

_ Yes  _ No 

a.  If  No,  were  you  aware  of  such  discussions  and  deliberations? 

_ Yes  _ No 

b.  If  not  involved  in  planning  deliberations,  do  you  think  you  should  have  been? 

_ Yes  _ No 

(If  not  involved  or  not  aware,  go  to  questions  on  Stage  3) 

5.  Please  indicate  the  degree  to  which  the  following  activities  are(have)  occurring(ed)  during 
the  process  of  planning  for  course  revisions. 


CHANGE  ACTIVITIES 

Circle  the  cells  that  best  describe  the  planning  stage 

Identifying 
revisions:  Identify 
revisions  with  the 

selected  course  that 

need  to  be 

implemented  , - K 

1 - ✓ 

Issues/revisions  not 
identified  or  agreed 
upon  as  the  most 
critical  to  change 

> 

Issues/revisions 
somewhat  identified 
and  agreed  upon  as 
the  most  critical  to 
change 

Issues/revisions  fully  identified 
and  agreed  upon  as  the  most 
critical  to  change 

Identifying 

solutions: 

Determine  how  to 

make  the 
changes/revisions 
to  the  selected 
course;  agreement 
regarding 

solutions/changes/ 

strategies 

Proposed 
solutions/changes 
did  not  match  the 

issues  that  were 
identified;  no  or 
limited  agreement 
currently  on 
solutions/changes/ 
strategies 

Proposed 
solutions/changes 
somewhat  matched 
the  identified  issues; 
moderate  agreement 

on 

solutions/changes/ 

strategies 

Proposed  solutions/changes 
matched/were  appropriate  for 
the  identified  issues;  full 
agreement  on 

solutions/changes/strategies 

6.  When  did  the  planning  process  start?  (Month/year) _ 

7.  When  did  the  planning  process  stop  -  decisions  were  made  regarding  the  issues  and  types 

of  revisions?  (Month/year) _ 
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Stage  3:  Developing  Course  Revisions 


8.  Were  you  involved  in  developing  the  actual  revisions  to  the  course  (e.g.,  new  lesson  plans, 
exercises,  new  training  materials,  and  so  on)? 

_ Yes  _ No 

a.  If  No,  were  you  aware  of  the  status  of  revisions?  _ Yes  _ No 

b.  If  not  involved,  do  you  think  you  should  have  been  in  developing  course  revisions? 

_ Yes  _ No 

(If  not  aware  or  not  involved  in  developing  course  revisions,  go  to  questions  on  Stage  4.) 

9.  Please  indicate  the  degree  to  which  the  following  activities  are  (have)  occurring(ed)  during 
the  process  of  developing  the  course  revisions. 


CHANGE  ACTIVITIES 

Circle  the  cells  that  best  describe  the  development  stage 

Course  revisions:  Develop 
all  planning  materials  to 
implement  solutions  (e.g., 
course  materials,  POIs, 
lesson  plans) 

Have  not  yet 
developed  or 
developed  less 
than  70%  of  the 

course  materials 

Have  developed  drafts 
of  a  majority  (70-100%) 
of  the  course  materials 

Have  developed  and 
finalized  course  material 

Instructor  training: 

Develop  and  execute 
instructor  training 

Have  not  yet 
developed 
materials  for 
training 
instructors 

Have  developed  draft 
materials  for  instructor 
training 

Have  fully  developed 
instructor  training  and 
executed  an  instructor 
training  course 

10.  When  did  the  development  process  start?  (Month/year) _ 

11.  When  did  the  development  process  end? 

a.  Materials  completed  (Month/year) _ 

b.  Instructor  training/preparation  completed  (Month/year) _ 

12.  If  all  revisions  have  not  been  completed,  when  do  you  estimate  the  course  revisions  will  be 

completed?  (Month/year) _ 


B-10 


Stage  4:  Implementing/Executing  the  Revised  Course 


13.  Were  you  involved  (or  will  you  be  involved)  in  implementing/piloting/executing/monitoring 
the  revised  course  in  anyway?  (e.g.,  as  an  instructor,  as  course  monitor)? 

_ Yes  _ No 

a.  If  No,  were  you  aware  of  the  implementation/execution  of  the  new  course? 

_ Yes  _ No 

b.  If  not  involved,  do  you  think  you  should  have  been  in  course  implementation? 

_ Yes  _ No 

(If  you  did  not  implement  the  course  or  are  not  aware  of/involved  with  course  execution,  go 
to  the  questions  on  Stage  5.) 

14.  Please  indicate  the  degree  to  which  the  following  activities  are  (have)  occurring(ed) 
regarding  course  implementation. 


CHANGE  ACTIVITIES 

Circle  the  cells  that 

best  describe  the  implementation/execution  stage. 

Implement 

change/course  revisions: 

Execute  course  revisions 

>=> 

Have  not  yet 
implemented  the 
revised  course 

Have  completed  one 
iteration  of  the  revised 

course 

Have  completed  more 
than  one  iteration  of 

the  revised  course 

If  implemented,  modify 
course  from  feedback: 

Collect  obtain  feedback 
from  instructors  and/or 
students  and  further 
revise  the  course  _ k 

_  ■=> 

Have  not  collected 
feedback  regarding 
the  course  revisions 

Have  collected  course 

feedback  but  have  not 

made  additional 

revisions  to  the  course 

Have  collected  course 

feedback  and  have 

made  additional 

revisions  to  the  course 

15.  When  was  the  first  revised  course  implemented? 

From  (Month/year) _ To  (Month/year) _ 

16.  If  there  has  been  more  than  one  course  implementation/execution,  list  the  dates  of  each: 

From  (Month/year) _ To  (Month/year) _ 

From  (Month/year) _ To  (Month/year) _ 

17.  If  the  revised  course  has  not  been  implemented,  when  do  you  estimate  the  course  will 
start? 

From  (Month/year) _ To  (Month/year) _ 
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Stage  5.  Sustaining  the  Revised  Course 


By  sustainment,  we  mean  that  the  revised  course  has  been  implemented  at  least  3  times. 

18.  Has  the  revised  course  been  implemented  at  least  three  times? 

_ Yes  _ No 

a.  If  "No",  do  you  believe  you  will  be  involved  in  sustaining  course  changes  in  any  way? 

_ Yes  _ No  _ Possibly 

b.  If  “No,"  do  you  believe  you  should  be  involved  in  sustaining  course  changes  in  any  way? 

_ Yes  _ No  Possibly 

(If  you  are  not  involved  with  the  sustainment  stage,  go  to  question  on  the  next  page.) 

19.  If  the  revised  course  has  been  implemented  at  least  3  times,  have  you  been  involved  in  any 
sustainment  activities  (e.g.,  providing  continued  logistical  support,  monitoring  to  see  if  the 
desired  changes  are  still  being  implemented,  ensuring  new  instructors  understand  how  to 
implement  course  revisions,  checking  on  student  performance/outcomes,  addressing  issues 
that  will  improve  the  course)? 

Yes  No 


20.  If  you  are  (will  be)  involved  in  course  sustainment,  please  indicate  the  degree  to  which  the 
following  activities  are  (have)  occurring(ed)  regarding  course  sustainment? 


CHANGE  ACTIVITIES 

Circle  the  cell  that  best  describe  this  stage. 

Implement  Sustainment 
Activities 

Activities  to  ensure  desired 
changes  continue 

No  planning  has  occurred 
for  formal  sustainment 
actions  or  identifying 
units/individuals 
responsible  for  this  stage 

Appropriate  agencies 
involved  in 

sustainment  have 

been  identified  and 
plans  initiated 

Sustainment 

activities  are 
ongoing 
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All  participants  please  answer  the  next  question  regardless  of  your  role(s)  in  the  course  revisions. 


21.  We  are  interested  in  your  interactions  with  other  agencies  or  units  during  course  revisions 
and  implementation. 

Using  the  diagram  shown  below,  draw  arrows  to  indicate  these  interactions. 

•  If  the  interaction  was  typically  one-way  -  you  initiated  interactions  with  another  agency 
or  unit,  or  another  agency/unit  initiated  interactions  with  you,  draw  a  single-directional 
arrow. 

(Initiate  or  Sender)  ^  (Receiver). 

•  If  there  was  relatively  constant  communications  between  you  and  other  organization, 

draw  a  two-headed  arrow  ^  ^  (each  initiated  communications). 

•  If  your  interactions  were  primarily  within  your  own  unit,  just  circle  your  unit. 


Insert  School's  or  CoE's  organizational  chart  on  next  page. 
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DECISION-  MAKING  PROCESS  INTERVIEW  QUESTIONS 


Position _ [interviewer  write  in] 

If  you  are  unable  to  answer  a  question  based  on  your  experiences,  simply  indicate  that  you  are  not 
aware  of  what  happened. 

1.  Describe  the  nature  of  your  involvement  in  deciding  which  course  should  be  involved  in  SITA  2. 

2.  Would  you  characterize  the  decision  process  as  from  the  bottom  up  or  the  top  down? 

Bottom  up  Top  down 

3.  Who  else  provided  input  to  the  decision?  (duty  positions)  _ 

4.  Was  more  than  one  course  considered?  Yes  No 

a.  If  yes,  how  many? _ 

b.  Were  several  meetings  or  briefings  held  in  the  process  of  making  a  decision?  Yes  No 

5.  Was  it  relatively  easy  to  reach  a  final  decision  /  recommendation  (consensus  easily  achieved)  or 

were  there  strong  arguments  for  other  courses?  _ 

Easy  Strong  Arguments  for  others 


6. 


What  were  the  primary  reasons  for  selecting  this  course?  Probe 

a.  New  equipment  in  the  field  that  needed  updated  training  (describe) 

b.  A  change  in  doctrine  (describe) 

c.  Feedback  +from  the  field  regarding  needed  changes  to  improve  graduate  proficiency 
(describe  feedback) 

d.  Desire  to  include  technology  (describe) 

e.  To  address  a  training  gap  (describe  gap) 

f.  Course  was  partially  modified  before  SITA  2 

g.  Course  was  on  the  cycle  to  be  revised/updated 

h.  Costs  were  envisioned  as  relatively  low  (e.g.,  no  special  contracts,  equipment  required) 

i.  Manpower  required  to  make  the  change  were  not  substantial 

j.  Changes  were  needed  to  shorten  the  training  time 

k.  Other 


7.  What  were  the  primary  reasons  for  rejecting  other  courses  that  were  considered? 

8.  Who  made  the  final  decision  on  the  course?  Name  the  duty  position _ 
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PLANNING  AND  IDENTIFYING  SOLUTIONS  INTERVIEW  QUESTIONS 


Position _ (Filled  in  by  interviewer) 

You  stated  that  you  were  involved  with  planning  the  revisions  to  the  course.  We  will  now  be  asking 
more  specific  questions  about  your  role. 

In  the  planning  stage,  the  specific  desired  changes  to  the  course(s)  are  examined  and  determined, 
priorities  are  made,  potential  solutions/alternatives  by  which  the  desired  changes  could  be 
implemented  are  identified,  a  plan  for  achieving  the  desired  end  state  is  established,  etc. 

Again,  if  you  cannot  answer  a  question  because  of  your  experiences,  simply  indicate  that  you  are  not 
aware  of  what  happened. 

1.  How  early  was  your  involvement  with  the  planning  process? 

_ At  the  start 

_ When  about  l/3rd  of  the  planning  was  done 

_ When  about  half  the  planning  was  done 

_ In  the  last  half  of  the  planning  cycle 

In  retrospect,  was  this  the  best  time  for  you  to  be  involved  or  would  another  time  have  been 
better? _ 

2.  To  the  best  of  your  knowledge,  did  the  planning  occur  prior  to  developing  the  revisions,  or 
did  both  stages  occur  simultaneously,  that  is  in  conjunction  with  each  other? 

_ Planning  was  prior  to  development 

_ Planning  done  in  conjunction  with  developing  changes 

_ Planning  occurred  prior  to  development,  but  course  development  activities  impacted 

or  changed  the  plan  as  well 

_ Do  not  know 

Other _ 

3.  Describe  the  nature  of  your  involvement  in  planning  the  changes  to  the  course. 


4.  How  many  lessons  are  in  the  course?  _ 

5.  Was  the  plan  to  modify  all  of  these  or  only  some?  _ All  _ Some -how  many? 

6.  Were  priorities  assigned  to  the  modifications? 

7.  Who  else  was  involved  in  the  planning  process? 

a.  Number _ 

b.  Duty  positions _ 
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8.  What  guidance  did  you  follow  when  planning  the  course  changes? 

Probe:  SITA  1,  SITA2,  Command  Guidance,  the  TRADOC  ALM  Pam,  Other? _ 

9.  Who  approved  the  final  plan?  (duty  position) _ 

10.  Please  describe  how  you  interacted  with  developers  during  the  planning  stage. 

11.  Please  describe  how  you  interacted  with  instructors  during  the  planning  stage. 

12.  What  were  the  procedures  for  reporting  to  more  senior  leaders  on  the  status  of  the  course  revision 
plans? 


Possible  Course  Changes 

Next,  please  go  to  the  questionnaire. 

13.  In  order  for  us  to  get  a  "big  picture"  of  the  major  areas  in  which  changes  were  planned,  please 
examine  the  list  on  the  first  page  of  the  questionnaire.  Indicate  which  areas  apply  to  your  course. 
Also,  please  add  any  other  areas  in  which  major  changes  were  planned. 


Course  content  -  the  subject  matter 

Tailored  training  to  better  address  individual  differences 

Means  of  delivering  the  training/instruction  (e.g.,  add  simulations,  less  PowerPoint) 
Soldier  competencies  per  the  Army  Learning  Model 
Learner-centered  and/or  problem  solving  activities/exercises/scenarios 
Principles  of  learning  to  enhance  learning,  retention,  and  transfer  to  the  field 
Other 


Interviewer:  Do  not  progress  to  the  next  question  until  the  individual  has 
completed  the  checklist  in  the  questionnaire. 
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14.  Now  we  would  like  you  to  consider  the  planning  process  for  the  course  changes.  Please  turn  to  the 
second  page  of  your  questionnaire  and  indicate  on  the  six  scales  how  the  changes  were  viewed  by 
your  team.  Your  rating  should  reflect  all  the  planned  changes. 


Interviewer:  Wait  until  the  individual  has  completed  the  ratings  prior  to  progressing  to  the  next  questions. 


Dimension 

Degree  of  change 


Circle  the  cell  that  best  describes  each  dimension 


Difficulty  of  making 
the  changes/course 


Change  in  Instructor 
behavior/procedures 


Time  requirement 


Manpower 

requirement 


Workload 


Minor  course  revisions: 
slight  adjustments  to 
the  current  lesson  plans 


Simple  tasks  to  make 
changes/course 
revisions  (e.g.,  low 
fidelity,  no  content 
change) 

Minor  and  few  changes 
needed  in  instructional 
techniques  from  the 
prior  course;  no  new 
instructor  training  (e.g., 
removal  of  PowerPoint 
slides,  more  repetition 
of  exercises  to  enhance 
learning  and  retention) 


No  additional  time 
required  to  make  the 
changes/course 


Changes  can  be/were 
made  with  current 
personnel  (e.g.,  1-2 
people) 


Workload  of  individuals 
remained  the  same 
during  the  planning 
process 

Inexpensive  to  make 
changes/course 
revisions  (e.g.,  no  new 
costs  to  make  changes) 


Moderate  course  revisions: 
Additions  or  revisions  of 
student  exercises,  slight 
changes  to  instructor 
training,  etc. 

Moderately  difficult  tasks  to 
make  changes/course 
revisions  (e.g.,  changes  to 
current  content,  changes  to 
existing  software) 

Somewhat  difficult  for  the 
instructor.  Most  desired 
changes  built  upon 
instructional  techniques 
previously  used.  Instructor 
training  needed  for  some 
blocks  of  instruction  (e.g., 
desk-top  simulations  or 
computer-based  training  to 
virtual  simulations,  insertion 
of  more  complex  problem¬ 
solving  exercises). 

Moderate  amount  of  time 
needed  to  make 
changes/course  revisions 
(e.g.,  a  few  hours  a  week) 


Some  additional  manpower 
was  needed  to  make  the 
course  changes/revisions 
(e.g.,  accomplished  changes 
by  adjusting  personnel  in- 
house) 

Workload  of  individuals 
increased  moderately  during 
the  planning  process. 

Somewhat  costly  to  make 
changes/course  revisions 
(e.g.,  required  some 
additional  costs  to  make 
planned  changes/course 
revisions) 


Large-scale  course  revisions: 
Re-write  of  the  POI  and 
lesson  plans,  extensive 
instructor  training 

Extremely  difficult/very 
complex  to  make 
changes/course  revisions 
(e.g.,  new  software 
development,  all  new 
content/lesson  plans) 
Substantial  changes  in 
instructional  techniques; 
training  required  to  prepare 
instructors  for  most  of  the 
course  (e.g.,  high  direct 
instruction  to  primarily 
facilitator,  focus  on  technical 
skills  only  to  also  focusing  on 
decision-making) 


Extreme  amount  of  time 
needed  to  make 
changes/course  revisions 
(e.g.,  entire  days  per  week 
spent  making  revisions) 


Much  additional  manpower 
needed  to  make  the 
changes/course  revisions 
(e.g.,  had  to  hire  outside 
personnel  with  specific  skills 
to  accomplish  changes  such 
as  software  developers) 
Workload  of  individuals 
increased  substantially 
during  the  planning  process. 

Very  costly  to  make 
changes/course  revisions 
(e.g.,  required  much 
additional  funds  to  make  the 
changes/course  revisions) 
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The  next  questions  are  on  specific  types  of  changes  that  could  have  been  planned  for  your  course.  If  a 
change  was  not  planned,  let  us  know.  Similarly  if  you  are  not  aware  of  some  details  of  the  planning 
process,  simply  indicate  that  you  are  not  aware. 

Course  Content  Questions 

15.  Did  the  plan  involve  any  modifications  to  the  course  content/subject  matter  (e.g.,  deletion  of 
content,  addition  of  new  content,  updating  of  content)?  Yes  No 

If  "yes",  explain  why  changes  were  perceived  as  desirable  or  needed.  _ 

16.  Also  please  tell  us  exactly  what  subject  matter  or  content  was  planned  to  be  changed  and  the 
number  of  lessons  identified  to  be  changed  in  the  plan. 

a.  Why _ 

b.  Content  to  be  changed _ 

c.  Number  of  lessons  affected 


Tailoring  Training 

One  of  the  concepts  stressed  in  the  Army  Learning  Model  is  that  "learning  should  be  tailored  to  the 
individual  learner's  experience  and  competence  level  based  on  the  results  of  a  pre-test  and/or 
assessment." 

17.  Did  the  plan  include  suggestions  to  instructors  or  training  developers  for  how  to  account  for 
differences  in  Soldier  competence  or  experience  (i.e.,  tailoring  training)?  Yes  No 

a.  Who  made  this  decision?  (duty  position)  _ 

b.  Did  the  plan  include  developing  tests  of  knowledge  as  a  means  of  identifying  critical 
individual  differences?  Yes  No 

If  yes,  was  the  intent  to  identify  individuals  who  might  need  assistance  OR  those 
who  needed  to  be  challenged  more? 

c.  Were  there  other  procedures  in  the  plan  for  addressing  individual  differences  and 
tailoring  the  instruction? 

Yes  No 

If  Yes,  describe  the  procedures 
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Methods  of  Instructional  Delivery 


18.  The  Army  Learning  Model  cites  many  methods  by  which  courses  can  be  conducted.  Please  complete 
the  questions  on  next  page  of  the  questionnaire  on  Methods  of  Instructional  Delivery.  Indicate  the 
changes  included  in  the  plan  for  your  course. 


Interviewer:  After  the  individuals  have  completed  the  questions,  IF  the  plan  was  to  insert  any 
new  technology,  have  them  indicate  what  the  technology  was  to  replace.  Probe  to  determine 
why  these  changes  were  planned. 

Also  if  they  revised  training  device  exercises  or  prerequisite  lessons,  probe  to  determine  the 
type  of  modification  planned. 


Methods  of  Instructional 
Delivery 

In  course 
prior  to 
revision? 

Planned 

to 

change? 

What  type  of  change  was 
planned  -  the  intent? 

Comments 

Instructor  Led 

Y  N 

Y  N 

More  /  Less 

Power-point  slides 

Y  N 

Y  N 

More  /  Less 

Hands-on  exercises  with 
equipment 

Y  N 

Y  N 

More  /  Less 

Problem-solving  exercises 
(not  technology-based) 

Y  N 

Y  N 

More  exercises 

More  with  increased  difficulty 
Fewer 

Describe  content 

variations. 

Technology-based 

Blended  Learning 

Y  N 

Y  N 

New  /  revised- more/ less 

If  new,  what  was  the 
technology  to  replace? 

Simulations/  Gaming 

Y  N 

Y  N 

New/  revised- more/ less 

Mobile  applications 

Y  N 

Y  N 

New/  revised- more /  less 

CBT/IMI  lessons 

Y  N 

Y  N 

New/  revised- more/ less 

Intelligent  tutors 

Y  N 

Y  N 

New/  revised- more/ less 

Training  devices 

Y  N 

Y  N 

New/  revised- more/ less 

Prerequisite  lessons  prior 
to  course  (distance 
learning) 

Y  N 

Y  N 

New/  revised- more/ less 

Other 

19.  Who  made  the  decisions  for  these  changes  (duty  position)?  _ 

20.  Was  there  a  consensus  on  the  planned  changes  which  you  just  marked?  Yes  No 

If  there  was  little  to  no  agreement,  how  was  an  agreement  reached? 

21.  What  were  the  reasons  of  inclusion,  rejection,  etc.? 
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Soldier  Competencies 


The  Army  Learning  Model  states  that  a  major  learning  outcome  is  to  develop  Soldier  competencies. 
These  competencies  are  listed  on  next  page  of  the  questionnaire. 

22.  Please  check  the  Soldier  competencies  selected  for  the  revised  course 


Check  if 

selected 

Competencies 

Tactical  and  technical  competence 

Character  and  accountability 

Comprehensive  fitness 

Adaptability  and  Initiative 

Teamwork  and  collaboration 

Lifelong  learner  (includes  digital  literacy) 

Communication  and  engagements  (oral,  written,  negotiation) 

Critical  thinking  and  problem  solving 

Cultural  and  joint  interagency,  intergovernmental,  and  multinational 
competence 

Interviewer.  After  the  individual  has  completed  marking  the  competencies  on  the 
questionnaire,  ask  the  following  questions. 


23.  Why  were  these  selected? _ 

24.  Why  were  the  others  rejected? _ 

25.  Was  there  a  consensus  on  which  to  include? _ 

26.  Who  made  these  decisions  (duty  position)  and  when? _ 

27.  What  was  strategy  or  plan  for  how  the  selected  competencies  could  be  integrated  into  the  revised 
course? 
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ALM  Concepts 


The  next  questions  are  on  other  concepts  in  the  Army  Learning  Model  which  you  may  have  incorporated 

in  your  revised  course. 

28.  Did  the  plan  include  revising  course  outcome  statements?  Yes  No 

a.  If  so,  were  these  outcome  statements  revised  during  the  planning  stage  or  during 
development  of  the  revised  lesson  plans  /  POI? 

Planning  stage  Development  stage 

b.  Who  developed  the  course  outcome  statements?  (Duty  position) _ 

29.  The  Army  Learning  Model  stresses  creating  learner-centered  activities.  Please  describe  the  plan  for 

these  activities. _ 

30.  The  Army  Learning  Model  also  stresses  the  importance  of  problem-solving  events  and  activities 

which  foster  the  ability  to  solve  problems,  to  think,  to  adapt,  to  take  initiative,  etc.  What  was  the 
plan  for  achieving  these  outcomes?  _ 

31.  Did  the  plans  for  learner-centered  activities  and/or  problem-solving  events  include  changes  in  the 
way  the  instructor  interacted  with  students?  Yes  No 

a.  If  Yes -what  was  the  intent? _ 

b.  Did  you  envision  that  these  changes  in  the  instructors'  role  would  be  sufficiently 
substantial  that  instructors  would  require  instructor  training  or  special  preparation?  Yes 
No 

32.  Were  there  any  plans  for  making  the  training  more  relevant  to  the  field,  to  the  Soldier's  future  duty 
assignment?  Yes  No 

If  so,  please  describe 

33.  Were  there  any  plans  for  changing  or  enhancing  feedback  to  the  students?  For  example,  AARs  or, 
feedback  on  simulation  exercises? 

a.  AARs? _ 

b.  More  feedback  on  performance  due  to  more  exercises?  _ 

c.  More  individualized  feedback? _ 

34.  What  were  the  plans  for  changing  the  course  to  improve  retention  of  skills  and  concepts  covered  in 
the  course? 
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Manpower  and  Logistics 


35.  What  was  the  perceived  impact  of  the  plan  on  training  developers?  (e.g.,  obtain  additional  training 
developers  to  handle  the  work  load,  obtain  training  developers  with  new  skills;  provide  current 
developers  with  more  training/skills?)  Describe. 

36.  What  was  the  perceived  impact  of  the  plan  on  instructors?  Describe. 

37.  Overall,  what  were  the  plans  for  preparing  instructors  for  the  revised  courses? 

38.  Did  the  plans  cover  potential  logistical  impacts  of  the  changes?  If  additional  or  new  impacts,  what 
was  the  solution  on  how  these  impacts  could  be  addressed?  Describe. 

39.  Did  the  plans  alter  any  time  requirements  for  the  course?  Did  the  course  require  more  or  less  time 
overall?  Within  each  day?  Less  synchronous  and  more  asynchronous  time? 

40.  Interviewer:  review  the  manpower  and  time  requirement  responses  in  the  questionnaire.  Ask  the 
interviewees  to  further  explain  their  responses. 


Summary  Questions 

41.  What  did  you  learn  in  the  process  of  planning  for  course  revisions  and  identifying  potential 

solutions? _ 

42.  What  were  the  best  decisions  or  recommendations  you  (or  others)  made? _ 

43.  What  would  you  change  if  you  were  involved  in  planning  revisions  for  another  course? _ 

44.  What  other  comments  do  you  have  on  the  planning  process? _ 


B-22 


PARTICIPANT  QUESTIONNAIRE:  PLANNING  FOR  REVISIONS 


Possible  Course  Changes 

1.  Check  the  areas  in  which  changes  were  planned  for  the  course.  Also,  add  any  other 
areas  in  which  major  changes  were  planned. 


Course  content  -  the  subject  matter 

Tailored  training  to  better  address  individual  differences 

Means  of  delivering  the  training/instruction  (e.g.,  add  simulations,  less  PowerPoint) 
Soldier  competencies  per  the  Army  Learning  Model 
Learner-centered  and/or  problem  solving  activities/exercises/scenarios 
Principles  of  learning  to  enhance  learning,  retention,  and  transfer  to  the  field 
Other 
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Initial  Planning 

2.  How  were  changes  viewed  by  your  team  in  the  planning  process? 


Dimension 

Degree  of  change 


Difficulty  of  making 
the  changes/course 
revisions 


l=> 


Change  in  Instructor 
behavior/procedures 


l=> 


Time  requirement 


Manpower 

requirement 


Workload 

— ► 

Costs 

>=> 


Circle  the  cell  that  best  describes  each  dimension 


Minor  course 
revisions:  slight 
adjustments  to  the 
current  lesson  plans 


Moderate  course  revisions: 
Additions  or  revisions  of 
student  exercises,  slight 
changes  to  instructor  training, 
etc. 


Large-scale  course  revisions: 
Re-write  of  the  POI  and 
lesson  plans,  extensive 
instructor  training 


Simple  tasks  to  make 
changes/course 
revisions  (e.g.,  low 
fidelity,  no  content 
change) 


Moderately  difficult  tasks  to 
make  changes/course  revisions 
(e.g.,  changes  to  current 
content,  changes  to  existing 
software) 


Extremely  difficult/very 
complex  to  make 
changes/course  revisions 
(e.g.,  new  software 
development,  all  new 
content/lesson  plans) 


Minor  and  few 
changes  needed  in 
instructional 
techniques  from  the 
prior  course;  no  new 
instructor  training 
(e.g.,  removal  of 
PowerPoint  slides, 
more  repetition  of 
exercises  to  enhance 
learning  and 
retention) 


Somewhat  difficult  for  the 
instructor.  Most  desired 
changes  built  upon 
instructional  techniques 
previously  used.  Instructor 
training  need  for  some  blocks 
of  instruction  (e.g.,  desk-top 
simulations  or  computer-based 
training  to  virtual  simulations, 
insertion  of  more  complex 
problem-solving  exercises). 


Substantial  changes  in 
instructional  techniques; 
training  required  to  prepare 
instructors  for  most  of  the 
course  (e.g.,  high  direct 
instruction  to  primarily 
facilitator,  focus  on 
technical  skills  only  to  also 
focusing  on  decision¬ 
making) 


No  additional  time 
required  to  make  the 
changes/course 
revisions 


Moderate  amount  of  time 
needed  to  make 
changes/course  revisions  (e.g., 
a  few  hours  a  week) 


Extreme  amount  of  time 
needed  to  make 
changes/course  revisions 
(e.g.,  entire  days  per  week 
spent  making  revisions) 


Changes  can  be/were 
made  with  current 
personnel  (e.g.,  1-2 
people) 


Some  additional  manpower 
was  needed  to  make  the 
course  changes/revisions  (e.g., 
accomplished  changes  by 
adjusting  personnel  in-house) 


Much  additional  manpower 
was  needed  to  make  the 
changes/course  revisions 
(e.g.,  had  to  hire  outside 
personnel  with  specific  skills 
to  accomplish  the  changes 
such  as  software 
developers) 


Workload  of 
individuals  remained 
the  same  during  the 
planning  process 


Workload  of  individuals 
increased  moderately  during 
the  planning  process. 


Workload  of  individuals 
increased  substantially 
during  the  planning  process. 


Inexpensive  to  make 
changes/course 
revisions  (e.g.,  no  new 
costs  to  make 
changes) 


Somewhat  costly  to  make 
changes/course  revisions  (e.g., 
required  some  additional  costs 
to  make  planned 
changes/course  revisions) 


Very  costly  to  make 
changes/course  revisions 
(e.g.,  required  much 
additional  funds  to  make 
the  changes/course 
revisions) 


B-24 


Additional  Manpower  Questions 

3.  Please  respond  to  the  following  questions  regarding  manpower  requirements: 

a.  How  many  people  were  initially  involved  in  planning  the  revisions? _ 

b.  What  was  the  total  number  of  people  involved  in  planning  the  revisions? _ 

c.  Did  the  people  involved  in  the  initial  planning  process  have  the  appropriate  skills  to  plan 

for  the  revisions? _ Yes  _ No 

i.  If  no,  what  additional  skills  were  needed  to  plan  for  the  revisions? _ 


Methods  of  Instructional  Delivery 

4.  The  Army  Learning  Model  cites  many  methods  by  which  courses  can  be  conducted.  Please 
answer  the  questions  in  the  table  below  regarding  the  plans  for  changing  the  methods  of 
instructional  delivery  in  your  course. 

Circle  Y  or  N,  More  or  Less,  New  or  revised  as  appropriate.  If  "revised"  is  circled  in  the  column,  then 
circle  either  more  or  less  which  follows. 


Methods  of  Instructional 
Delivery 

In  course  prior 
to  revision? 

Planned  to 
change? 

What  type  of  change  was  planned  - 
the  intent? 

Instructor  led 

Y  N 

Y  N 

More  /  Less 

PowerPoint  slides 

Y  N 

Y  N 

More  /  Less 

Hands-on  exercises  with 

Y  N 

Y  N 

More  /  Less 

equipment 

Problem-solving  exercises 

_ More  exercises 

(not  technology-based) 

Y  N 

Y  N 

_ More  with  increased  difficulty 

_ Fewer 

Technology-based: 

Blended  Learning 

Y  N 

Y  N 

New  /  revised-  more/  less 

Simulations/  Gaming 

Y  N 

Y  N 

New  /  revised-  more/  less 

Mobile  applications 

Y  N 

Y  N 

New  /  revised-  more  /  less 

CBT/IMI  lessons 

Y  N 

Y  N 

New  /  revised-  more/  less 

Intelligent  tutors 

Y  N 

Y  N 

New  /  revised-  more/  less 

Training  devices 

Y  N 

Y  N 

New  /  revised-  more/  less 

Prerequisite  lessons  prior  to 

Y  N 

Y  N 

New  /  revised-  more/  less 

course  (distance  learning) 

Other 
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Soldier  Competencies 


5.  Please  respond  if  the  competency  was  selected  for  integration  into  the  revised  course. 


Check  (x)  if  the 
competency  was 
selected  for  integration 
into  the  revised  course 

Soldier  Competencies 

Tactical  and  technical  competence 

Character  and  accountability 

Comprehensive  fitness 

Adaptability  and  Initiative 

Teamwork  and  collaboration 

Lifelong  learner  (includes  digital  literacy) 

Communication  and  engagements  (oral,  written,  negotiation) 

Critical  thinking  and  problem  solving 

Cultural  and  joint  interagency,  intergovernmental,  and  multinational 
competence 
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DEVELOPING  COURSE  REVISIONS  INTERVIEW  QUESTIONS 


Position _ (Filled  in  by  interviewer) 

You  stated  that  you  were  involved  with  developing  or  monitoring/supervising  course  revisions.  We  will 
now  ask  more  specific  questions  about  your  role. 

If  you  cannot  answer  a  question  because  of  your  experiences,  simply  indicate  that  you  are  not  aware  of 
what  happened. 

1.  To  the  best  of  your  knowledge,  did  the  planning  occur  prior  to  developing  the  revisions,  or 
did  both  stages  occur  simultaneously,  that  is  in  conjunction  with  each  other? 

_ Planning  was  prior  to  development 

_ Planning  done  in  conjunction  with  developing  changes 

_ Planning  occurred  prior  to  development,  but  course  development  activities  impacted  or 

changed  the  plan  as  well 

_ Do  not  know 

Other 


2.  Were  you  informed  of  the  plan?  Yes  No 

3.  When  we  talk  about  developing  course  revisions,  we  assume  developers  were  provided 
guidance  from  those  who  did  the  planning  and/or  from  key  decision-makers.  We 
understand  that  guidance  could  have  been  general  or  very  specific.  But  we  assume  that  you 
understood  the  scope  of  your  responsibilities  and  the  product  that  decision-makers  and 
planners  desired. 

Describe  the  nature  of  your  involvement  in  developing  changes  to  the  course. _ 


Interviewer: 

The  first  set  of  general  questions  which  follows  is  for  training  developers. 
The  second  set  is  for  course  monitors/managers. 

The  third  set  is  for  both  training  developers  and  course  monitors/managers. 
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Training  Developers 


1.  When  you  made  revisions  to  the  course,  how  many  lessons  did  you  work  on? 

2.  Who  else  was  involved  in  the  development  process? 

a.  Number _ 

b.  Duty  positions  _ 

c.  Did  you  work  collaboratively  with  others  or  primarily  on  your  own? 


3.  What  guidance  did  you  follow  when  making  the  course  changes?  Probe: 

a.  SITA1 

b.  SITA2 

c.  Command  Guidance 

d.  TRADOC  ALM  Pam 

e.  Course  manager 

f.  Other? 


4.  Did  you  use  the  Training  Development  Capability  (TDC)  to  design  and  develop  the  course 
revisions? 

_ Yes  _ No 

a.  If  yes,  can  you  describe  how  you  used  the  tool  to  design  and  develop  the  course 
revisions? 

b.  What  is  your  perception  of  the  tools'  capabilities  in  meeting  the  requirements  for  the 
course  revisions? 

5.  Did  you  receive  any  special  training  to  enable  you  to  work  more  efficiently  or  better  on 

changes  to  this  course?  _ Yes  _ No 

If  yes,  describe  the  training 

6.  How  much  total  time  do  you  estimate  you  spent  making  revisions  to  the  course? 

7.  Were  there  any  unexpected  costs? 
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Course  Monitors/Managers 


1.  When  you  monitored  the  development  process,  how  many  lessons  were  revised? 

2.  How  many  individuals  were  involved  in  the  development  process? 

3.  Was  this  adequate  to  handle  the  workload?  Yes  No 

a.  If /Vo,  were  you  able  to  obtain  additional  individuals?  Yes  No 

4.  Did  each  person  have  the  same  degree  of  responsibility,  or  were  a  few  individuals 
responsible  for  most  of  the  changes? 

5.  Did  the  training  developers  working  on  this  course  have  the  appropriate  skill  sets  to 
accomplish  the  changes?  Yes  No 

a.  If  Yes,  what  were  these  skill  sets? 

b.  If  /Vo  -  not  all  skills  existed,  what  other  skills  were  required  or  needed  to  be 
improved? 

i.  How  was  this  problem  resolved? 

ii.  Was  additional  training  provided  to  the  developers  or  were  you  able  to 
obtain  others  with  the  needed  skills? 

6.  What  guidance  did  you  (or  others)  give  to  developers?  _ 

7.  Did  your  involvement  in  the  course  changes  impact  your  workload?  Yes  No 

If  yes,  please  describe. 
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Training  Developers  and  Course  Monitors/Managers 


Interviewer:  The  next  questions  should  be  asked  of  both  developers  and  individuals  who 
monitored  the  course  development  process. 

1.  Please  describe  how  you  interacted  with  planners  during  the  development  stage. 

2.  Please  describe  how  you  interacted  with  instructors  during  the  development  stage. 

3.  What  were  the  procedures  for  reporting  to  more  senior  leaders  on  the  status  of  the  course 
revisions? 

Possible  Course  Changes 

4.  In  order  for  us  to  understand  the  major  areas  in  which  changes  or  revisions  were  made  in 
the  course,  please  examine  the  list  in  the  questionnaire  and  indicate  the  areas  in  which  you 
were  involved.  Also,  please  indicate  any  other  major  areas  in  which  you  worked  or 
monitored  during  the  development  process. 

Interviewer: 

•  Have  developers  answer  questions  A  and  B  only. 

•  Have  course  monitors/managers  answer  questions  A  and  C  only. 

•  Indicate  that  "DNK"  stands  for  "do  not  know"  whether  an  area  was  in  the  plan. 

•  Do  not  progress  to  the  next  question  until  the  checklist  is  completed. 


AREA 

Developers  and 
Monitors 

Developers 

Only 

Monitors 

Only 

A)  Was  this  area 
in  the  plan? 

B)  Did  you  work 
in  this  area? 

C)  Did  you 
monitor  work  in 

this  area? 

a.  Course  outcome  statements 

Y  N  DNK 

Y  N 

Y  N 

b.  Soldier  competencies  per  the  Army 
Learning  Model 

Y  N  DNK 

Y  N 

Y  N 

c.  Course  content  -  the  subject  matter 

Y  N  DNK 

Y  N 

Y  N 

d.  Tailored  training  to  better  address 
individual  differences 

Y  N  DNK 

Y  N 

Y  N 

e.  Means  of  delivering  the 
training/instruction  (e.g.,  add  simulations, 
less  PowerPoint) 

Y  N  DNK 

Y  N 

Y  N 

f.  Learner-centered  and/or  problem 
solving  activities/exercises/scenarios 

Y  N  DNK 

Y  N 

Y  N 

g.  Principles  of  learning  to  enhance 
learning,  retention,  and  transfer  to  the 
field 

Y  N  DNK 

Y  N 

Y  N 

h.  Other 

Y  N  DNK 

Y  N 

Y  N 
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5.  As  a  developer  or  an  individual  monitoring/responsible  for  course  revisions,  please  indicate 
on  the  7  scales  in  the  questionnaire  how  the  changes  were  viewed  by  your  team.  Your 
rating  should  reflect  all  the  planned  changes  of  which  you  were  aware. 

Interviewer:  Do  not  progress  to  the  next  questions  until  the  ratings  are  complete. 


Dimension 


Circle  the  cell  that  best  describes  each  dimension 


Degree  of 
change 


► 


Minor  course  revisions: 
slight  adjustments  to 
the  current  lesson  plans 


Difficulty  of 
making  the 
changes/course 
revisions 


Simple  tasks  to  make 
changes/course 
revisions  (e.g.,  low 
fidelity,  no  content 
change 


Moderate  course  revisions: 
Additions  or  revisions  of 
student  exercises,  slight 
changes  to  instructor  training, 
etc. 

Moderately  difficult  tasks  to 
make  changes/course 
revisions  (e.g.,  changes  to 
current  content,  changes  to 
existing  software) 


Change  in 
Instructor 
behavior/ 
procedures 


Time 

requirement 


Manpower 

requirement 


Minor  and  few  changes 
needed  in  instructional 
techniques  from  the 
prior  course;  no  new 
instructor  training  (e.g., 
removal  of  PowerPoint 
slides,  more  repetition  of 
exercises  to  enhance 
learning  and  retention) 


No  additional  time 
required  to  make  the 
changes/course 
revisions 

Changes  can  be/were 
made  with  current 
personnel  (e.g.,  1-2 
people) 


Somewhat  difficult  for  the 
instructor.  Most  desired 
changes  built  upon 
instructional  techniques 
previously  used.  Instructor 
training  needed  for  some 
blocks  of  instruction  (e.g., 
desk-top  simulations  or 
computer-based  training  to 
virtual  simulations,  insertion 
of  more  complex  problem¬ 
solving  exercises). 

Moderate  amount  of  time 
needed  to  make 
changes/course  revisions 
(e.g.,  a  few  hours  a  week) 
Some  additional  manpower 
was  needed  to  make  the 
course  changes/revisions 
(e.g.,  accomplished  changes 
by  adjusting  personnel  in- 
house) 


Workload 


Workload  of  individuals 
remained  the  same 
during  the  planning 
process 

Inexpensive  to  make 
changes/course 
revisions  (e.g.,  no  new 
costs  to  make  changes) 


Workload  of  individuals 
increased  moderately  during 
the  planning  process. 


Somewhat  costly  to  make 
changes/course  revisions 
(e.g.,  required  some 
additional  costs  to  make 
planned  changes/course 
revisions) 


Large-scale  course  revisions:  Re¬ 
write  of  the  POI  and  lesson  plans, 
extensive  instructor  training 


Extremely  difficult/very  complex 
to  make  changes/course  revisions 
(e.g.,  new  software  development, 
all  new  content/lesson  plans) 


Substantial  changes  in 
instructional  techniques;  training 
required  to  prepare  instructors 
for  most  of  the  course  (e.g.,  high 
direct  instruction  to  primarily 
facilitator,  focus  on  technical  skills 
only  to  also  focusing  on  decision¬ 
making) 


Extreme  amount  of  time  needed 
to  make  changes/course  revisions 
(e.g.,  entire  days  per  week  spent 
making  revisions) 

Much  additional  manpower  was 
needed  to  make  the 
changes/course  revisions  (e.g., 
had  to  hire  outside  personnel 
with  specific  skills  to  accomplish 
the  changes  such  as  software 
developers) 

Workload  of  individuals  increased 
substantially  during  the  planning 
process. 


Very  costly  to  make 
changes/course  revisions  (e.g., 
required  much  additional  funds  to 
make  the  changes/course 
revisions) 
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The  next  questions  are  on  specific  types  of  changes  that  could  have  been  made  in  your  course.  If  a 
change  was  not  made,  let  us  know.  Similarly  if  you  are  not  aware  of  some  details  of  the  development 
process,  simply  indicate  that  you  are  not  aware. 


•  Interviewer  (general  guidance):  If  an  individual  did  NOT  have  a  role  in  making/=  or  monitoring 
a  specific  type  of  change,  progress  to  the  next  set  of  questions. 

•  As  both  developers  and  individuals  monitoring/responsible  for  the  changes  will  be 
interviewed,  the  words  "monitored"  or  "monitoring"  have  been  parenthetically  inserted  in 
some  questions  to  use  when  interviewing  these  individuals. 

•  There  are  other  places  in  the  interview  where  questions  are  phrased  differently  for  developers 
and  monitors  or  where  separate  sets  of  questions  are  asked. 


Course  Outcome  Statements 


6.  Did  you  participate  in  generating  (or  monitoring)  course  outcome  statements?  Yes  No 

a.  If  Yes,  please  describe  the  process  by  which  this  was  accomplished. 

b.  If /Vo,  please  indicate  who  did,  if  known  to  you  (duty  position) _ 

c.  Were  you  provided  the  course  outcome  statements  prior  to  any  development 
activity?  Yes  No 

Soldier  Competencies 

The  Army  Learning  Model  states  that  a  major  learning  outcome  is  to  develop  Soldier  competencies. 

Please  go  to  the  questionnaire  and  complete  the  questions  on  Soldier  Competencies. 
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7.  Have  developers  answer  questions  A  and  B.  Have  course  monitors/managers  answer 
questions  A  and  C. 


Developers/ 

Monitors 

Developers  Only 

Monitors  Only 

COMPETENCY 

A)  Was  this 
competency 
in  the  plan? 

B)  Did  you  work  on 
integrating  this  competency 
into  the  course? 

C)  Did  you  monitor  work 
integrating  this  competency 
into  the  course? 

a.  Tactical/Technical 
Competence 

Y  N  DNK 

Y  N 

Y  N 

b. 

Character/Accountability 

Y  N  DNK 

Y  N 

Y  N 

c.  Comprehensive  Fitness 

Y  N  DNK 

Y  N 

Y  N 

d.  Adaptability/Initiative 

Y  N  DNK 

Y  N 

Y  N 

e. 

Teamwork/Collaboration 

Y  N  DNK 

Y  N 

Y  N 

f.  Lifelong  learner 
(includes  digital  literacy) 

Y  N  DNK 

Y  N 

Y  N 

g.  Communication/ 
Engagements  (oral, 
written,  negotiation) 

Y  N  DNK 

Y  N 

Y  N 

h.  Critical  thinking/ 

Problem  solving 

Y  N  DNK 

Y  N 

Y  N 

i.  Cultural/Joint 
interagency, 
intergovernmental,  and 
multinational  competence 

Y  N  DNK 

Y  N 

Y  N 

8.  How  were  lesson  plans/activities  created  or  modified  to  integrate  competencies  into  the 
course?  Please  describe  for  each  competency  you  identified  in  the  questionnaire. 

9.  What  challenges  were  encountered  when  integrating  competencies  in  the  lesson  plans? 

10.  Were  student  measures  or  assessments  of  these  competencies  developed?  Yes  No 

a.  If  Yes,  describe  the  nature  of  the  test/assessment. 

b.  Can  you  share  a  specific  example  with  us? 

11.  Who  reviewed  these  products;  revised/new  lesson  plans?  (duty  position) _ 

12.  Who  approved  these  products;  revised/new  lesson  plans?  (duty  position) _ 
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Course  Content 


Interviewer:  Ask  the  following  questions  only  of  individuals  who  indicated  they  were 
involved  with  course  content  changes. 


13.  You  indicated  you  were  involved  with  changes  to  the  course  content,  the  subject  matter. 
Tell  us  exactly  what  subject  matter  or  content  was  changed  and  the  number  of  lessons 
identified  to  be  changed  in  the  plan. 

a.  Why _ 

b.  Content  to  be  changed _ 

c.  #  of  lessons  affected _ 

14.  Describe  what  was  involved  in  making  changes  to  the  content.  Probe: 

a.  Research  new/appropriate  material 

b.  Talk  to  SMEs 

c.  Generate  new  lesson  plans  with  new  exercises,  instructional  techniques,  etc. 

d.  Develop  supporting  materials  -based  on  technology  or  non-based  on  technology 

e.  Develop  new  tests  for  assessing  Soldier  performance 

f.  Other _ 

15.  How  was  the  new  content  or  subject  matter  validated? 

16.  How  much  time  did  this  validation  require? 

17.  Who  reviewed  the  revised/new  lesson  plans?  (duty  position) _ 

18.  Who  approved  the  revised/new  lesson  plans?  (duty  position) _ 

Tailoring  Training 

Interviewer:  Ask  the  following  questions  only  of  individuals  who  indicated  they  were  involved 
with  some  aspect  of  tailored  training  (tailoring  the  course  to  the  individual  learner's  experience 
and  competence  level  based  on  the  results  of  a  pre-test  and/or  assessment). 


19.  Did  you  create  [or  monitor  the  creation  of]  "tests"  or  other  assessments  to  identify  critical 
individual  differences  related  to  specific  blocks  of  instruction  (e.g.,  prior  knowledge 
regarding  subject  matter)?  Yes  No 

If  Yes,  for  which  blocks  of  instruction  did  you  create  the  tests? 

20.  Did  you  develop  any  support  materials  or  instructional  techniques  intended  to  facilitate  the 
instructor's  ability  to  tailor  training  [or  monitor  this  process]?  Yes  No 
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If  Yes,  describe? 


21.  How  many  developers  were  involved  in  these  changes? 

22.  How  much  time  was  required? 

a.  Was  this  time  sufficient? 

23.  What  challenges  did  this  change  present? 

24.  Who  reviewed  the  work  on  tailoring  the  training?  (duty  position) _ 

25.  Who  approved  the  work  on  tailoring  the  training?  (duty  position) _ 

Methods  of  Instructional  Delivery 

Interviewer:  The  Army  Learning  Model  cites  many  methods  by  which  courses  can  be  conducted. 
Please  examine  the  table  in  the  questionnaire  and  answer  the  questions  on  methods  of  instructional 
delivery.  List  any  other  methods  that  were  changed  as  well. 
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26.  Training  developers  answer  questions  A  and  B.  Monitors  answer  questions  A  and  C. 


Developers  &  Monitors 

Developers  Only 

Monitors  Only 

METHODS  OF 

INSTRUCTIONAL  DELIVERY 

A)  Was  this  area  in  the 
plan? 

B)  Did  you  work 
on  this  area? 

C)  Did  you  monitor 
this  area? 

a.  Instructor  led 

Y  N  DNK 

Y  N 

Y  N 

b.  PowerPoint  slides 

Y  N  DNK 

Y  N 

Y  N 

c.  Hands-on  exercises  with 
equipment 

Y  N  DNK 

Y  N 

Y  N 

d.  Problem-solving  exercises 
(not  technology-based) 

Y  N  DNK 

Y  N 

Y  N 

e.  Training  devices 

Y  N  DNK 

Y  N 

Y  N 

f.  Prerequisite  lessons  prior 
to  course  (distance  learning) 

Y  N  DNK 

Y  N 

Y  N 

Technology-based 

g.  Blended  Learning 

Y  N  DNK 

Y  N 

Y  N 

h.  Simulations/  Gaming 

Y  N  DNK 

Y  N 

Y  N 

i.  Mobile  applications 

Y  N  DNK 

Y  N 

Y  N 

k.  CBT/IMI  lessons 

Y  N  DNK 

Y  N 

Y  N 

1.  Intelligent  tutors 

Y  N  DNK 

Y  N 

Y  N 

m.  Other 

Y  N  DNK 

Y  N 

Y  N 

Interviewer:  The  first  set  of  questions  on  Methods  of  Instructional  Delivery  is  for  Training 
Developers  and  the  second  set  of  questions  is  for  Course  Monitor/Managers. 


Training  Developers  Only:  (Methods  of  Instructional  Delivery) 

27.  Describe  what  you  did  in  each  area  you  marked  on  the  questionnaire  (e.g.,  develop 
simulation  exercises",  create  prerequisite  lessons  ...). 

a.  Were  you  responsible  for  revising  the  associated  lesson  plans  as  well?  Yes  No 

b.  Probe  to  get  a  good  picture  of  responsibilities  and  the  process. 

28.  If  the  plan  included  the  generation  of  new  scenarios/exercises,  can  you  describe  a  few  of 
these  that  were  created? 

a.  How  did  you  obtain  the  content  to  develop  the  scenarios/exercises? 

b.  How  was  the  content  validated? 

29.  Did  you  work  with  someone  else  when  making  these  changes?  _ 

30.  How  much  time  was  required?  _ 

a.  Was  this  time  sufficient?  _ 

31.  What  challenges  did  these  changes  present?  _ 
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32.  Who  reviewed  your  products;  revised/new  lesson  plans?  (duty  position) _ 

33.  Who  approved  your  products;  revised/new  lesson  plans?  (duty  position) _ 

Monitors  or  Course  Managers  Only  (Methods  of  Instructional  Delivery) 

34.  Describe  your  role  in  each  of  the  areas  you  just  identified  on  the  questionnaire. 

Both  Developers  and  Monitors -Army  Learning  Concepts 

35.  The  next  questions  are  on  other  concepts  in  ALM  which  may  have  been  incorporated  in  the 
revised  course.  Examine  the  areas  listed  in  the  questionnaire  and  indicate  your 
involvement.  Developers  answer  A  and  B.  Monitors  answer  B  and  C. 


AREA 

Developers/ 

Monitors 

Developers 

Only 

Monitors 

Only 

A)  Was  this  area 
in  the  plan? 

B)  Did  you  work 
in  this  area? 

C)  Did  you  monitor 
work  in  this  area? 

a.  Learner-centered  activities 

Y  N  DNK 

Y  N 

Y  N 

b.  Problem-solving  events  and 
activities  (foster  ability  to  solve 
problems,  to  think,  to  adapt,  to 
take  initiative,  etc.) 

Y  N  DNK 

Y  N 

Y  N 

c.  Making  the  course  more 
relevant  to  the  field  (to  Soldier's 
future  duty  assignment) 

Y  N  DNK 

Y  N 

Y  N 

d.  Changing  or  enhancing 
feedback  to  students 

Y  N  DNK 

Y  N 

Y  N 

e.  Techniques/Procedures  to 
improve  the  retention  of  skills  or 
knowledge  of  course  content 

Y  N  DNK 

Y  N 

Y  N 

Interviewer:  If  the  individual  was  NOT  involved  in  a  specific  development  activity,  go  to  questions  on 
the  next  area. 

Learner-centered  Activities 


36.  Did  you  receive  any  guidance  other  than  the  MTTs  in  how  to  create  student-focused 
activities?  Yes  No 

37.  Can  you  provide  examples  of  learner-centered  learning  activities  that  you  created  (that 
were  created  by  developers )? 

38.  Were  there  models  from  other  courses  that  helped  in  developing  these  activities?  Yes  No 

39.  Did  the  learner-centered  activities  involve  a  major  change  in  the  way  the  instructor 
presented  the  instruction?  Yes  No 

a.  If  Yes,  please  describe. 
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40.  Did  any  of  the  techniques  you  developed  [monitored]  involve  what  is  called  scaffolding, 
whereby  the  amount  of  instructor  support  is  typically  provided  at  first,  then  modified  and 
gradually  removed  according  to  the  needs  of  the  learner?  Yes  No 

a.  Can  you  provide  any  examples  of  this? 

Problem-solving  Events  and  Activities 

41.  Did  you  receive  any  guidance  other  than  the  MTTs  in  how  to  create  problem-solving  events 
and  activities?  Yes  No 

42.  How  did  you  redesign  the  course  to  accomplish  this  intent?  [For  monitors,  ask:  how  was 
the  course  redesigned  to  accomplish  this  intent?] 

43.  Can  you  provide  examples  of  the  problem-solving  events  and  activities  that  you  (were) 
created? 

44.  Did  they  progress  from  basic  situations  to  more  complex  situations?  Yes  No 

45.  Did  you  design  (review)  any  lessons  or  activities  where  the  students  had  to  solve  problems 
on  their  own?  Yes  No 

46.  Were  there  any  models  from  other  courses  that  helped  in  developing  these  activities?  Yes 
No 

47.  What  challenges  did  you  face  in  developing  these  revised  lesson  plans?  [For  monitors  ask: 
what  challenges  did  training  developers  face  in  developing  these  revised  lesson  plans?] 

Learner-centered  Activities  and/or  Problem-solving  Events 

48.  Did  the  learner-centered  activities  and/or  problem-solving  events  on  which  you  worked 
[monitored]  impact  the  pattern  of  instructor-student  interaction?  Yes  No  Please  describe 

a.  Mode  of  instructor-student  interaction  (e.g.,  a  change  from  large  group  to  small 
group) 

b.  Peer-to-peer  instruction  or  team  based  interactions  with  less  direct  instructor 
guidance 

c.  More  independent  study/work/presentations  by  students 

d.  Instructor  role  changing  to  be  a  facilitator 

49.  After  the  lesson  plans/activities  were  developed,  did  you  view  the  changes  in  instructor- 
student  interaction  as  sufficiently  substantial  that  they  would  require  instructor  training? 

a.  If  yes,  please  describe  the  nature  of  the  instructor  training. 

Relevance  of  Course  to  the  Field  (to  Soldier's  future  duty  assignment) 
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50.  Please  describe  how  the  lessons  plans  were  changed  to  make  them  more  relevant  to  the 
field,  to  the  Soldier's  future  duty  assignment?  Please  provide  examples  of  this. 

Feedback  to  Students 

51.  In  what  ways  did  you  change  or  enhance  feedback  to  students  in  the  lesson  plans?  Please 
describe  the  type  of  feedback  techniques  you  incorporated  into  the  revised  lesson  plans. 
Probes: 

a.  AARs? 

b.  More  feedback  on  performance  due  to  more  exercises? 

c.  More  individualized  feedback? 

Improving  Retention  of  Skills  and  Knowledge 

52.  What  techniques,  activities,  or  procedures  did  you  incorporate  in  the  course  that  would 
improve  retention  of  skills  or  knowledge  of  the  concepts  and  principles  covered  in  the 
course?  Please  provide  some  examples. 

Skills  Necessary  to  Make  the  Course  Changes:  For  Training  Developers  Only 

53.  As  a  training  developer,  what  were  the  primary  skills  you  found  valuable  in  making  changes 
to  the  course? 

54.  Do  you  feel  that  changes  could  have  been  made  better  or  faster  if  you  had  additional  skills? 
If  so,  what  are  they? 

Resources:  For  Course  Monitors/Managers  Only 

55.  We  are  also  interested  in  your  estimates  of  the  resources  required  to  make  changes  in 
your  course.  On  the  last  two  pages  of  the  questionnaire,  we  ask  you  to  provide  estimates 
of  manpower  and  time  requirements.  Please  complete  it  based  on  your  experiences  with 
the  course  and  what  you  have  observed. 

For  Both  Training  Developers  and  Course  Monitors/Managers 

56.  Considering  the  course  revisions  that  were  made,  do  you  think  the  set  of  skills  and 
capabilities  required  of  training  developers  should  be  expanded  OR  do  you  think  a  team  is 
needed  with  the  appropriate  balance  of  skills?  Please  describe. 

57.  What,  if  any,  unexpected  logistical  issues  were  identified  during  the  process  of  revising  the 
course  and  developing/revising  lesson  plans? 

58.  How  are  these  lessons  learned  applicable  to  other  courses  that  may  be  revised  in  your 
school? 

59.  What  did  you  learn  in  the  process  of  revising  this  course  that  you  would  repeat? 
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60.  What  would  you  change? 


61.  What  were  the  best  products  you  produced  (or  that  were  produced)? 

62.  What  recommendations  do  you  have  for  other  schools  and/or  for  similar  courses? 

63.  Do  you  have  any  additional  comments  to  make  regarding  the  development  process? 
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PARTICIPANT  QUESTIONNAIRE:  DEVELOPING  REVISIONS 


1.  Duty  position:  _ 

2.  Major  Course  Changes 

All  individuals  answer  "A":  To  the  best  of  your  knowledge,  please  circle  your  response  to  indicate 
which  areas  were  addressed  in  the  plan  to  change/revise  the  course. 

Training  Developers  also  answer  "B":  Please  circle  your  response  to  indicate  the  major  areas  in  which 
you  developed  changes  to  the  course.  Describe  any  other  major  areas  in  which  you  worked. 

Course  Monitors  also  answer  "C":  Please  circle  your  response  to  indicate  the  major  areas  that  you 
monitored. 

The  "DNK"  under  "A"  stands  for  "Do  Not  Know"  -  that  is,  you  do  not  know  whether  the  area  was  in  the 
plan. 


AREAS  OF  REVISIONS/CHANGES 

Developers  and 
Monitors 

Developers 

Only 

Monitors 

Only 

A)  Was  this  area 
in  the  plan? 

B)  Did  you  work 
in  this  area? 

C)  Did  you  monitor 
work  in  this  area? 

a.  Course  outcome  statements 

Y  N  DNK 

Y  N 

Y  N 

b.  Soldier  competencies  per  the  Army 
Learning  Model 

Y  N  DNK 

Y  N 

Y  N 

c.  Course  content  -  the  subject  matter 

Y  N  DNK 

Y  N 

Y  N 

d.  Tailored  training  to  better  address 
individual  differences 

Y  N  DNK 

Y  N 

Y  N 

e.  Means  of  delivering  the 
training/instruction  (e.g.,  add 
simulations,  less  PowerPoint) 

Y  N  DNK 

Y  N 

Y  N 

f.  Learner-centered  and/or  problem 
solving  activities/exercises/scenarios 

Y  N  DNK 

Y  N 

Y  N 

g.  Principles  of  learning  to  enhance 
learning,  retention,  and  transfer  to  the 
field 

Y  N  DNK 

Y  N 

Y  N 

h.  Other:  ( please  write  in  area) 

Y  N  DNK 

Y  N 

Y  N 
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3.  How  were  changes  viewed  by  your  team  during  development? 


Dimension 

Circle  the  cell  that  best  describes  each  dimension 

Degree  of 
change 

■=> 

Minor  course 
revisions:  Slight 
adjustments  to  the 
current  lesson  plans 

Moderate  course  revisions: 

Additions  or  revisions  of 
student  exercises,  slight 
changes  to  instructor  training. 

Large-scale  course 
revisions:  Re-write  of  the 

POI  and  lesson  plans, 
extensive  instructor  training 

Difficulty  of 
making  the 
changes/cour 
se  revisions 

K 

Simple  tasks  to  make 
changes/course 
revisions  (e.g.,  low 
fidelity,  no  content 
change) 

Moderately  difficult  tasks  to 
make  changes/course 
revisions  (e.g.,  changes  to 
current  content,  changes  to 
existing  software} 

Extremely  difficult/very 
complex  to  make 
changes/course  revisions 
(e.g.,  new  software 
development,  all  new 
content/lesson  plans) 

V 

Change  in 
Instructor 
behavior/ 
procedures 

K 

Minor  and  few 
changes  needed  in 
instructional 
techniques  from  the 
prior  course;  no  new 
instructor  training 
(e.g.,  removal  of 
PowerPoint  slides, 
more  repetition  of 
exercises  to  enhance 
learning  and 
retention) 

Somewhat  difficult  for  the 

instructor.  Most  desired 
changes  built  upon 
instructional  techniques 
previously  used.  Instructor 
training  need  for  some  blocks 
of  instruction  (e.g.,  desk-top 
simulations  or  computer- 
based  training  to  virtual 
simulations,  insertion  of  more 
complex  problem-solving 
exercises). 

Substantial  changes  in 
instructional  techniques; 
training  required  to  prepare 
instructors  for  most  of  the 
course  (e.g.,  high  direct 
instruction  to  primarily 
facilitator,  focus  on 
technical  skills  only  to  also 
focusing  on  decision¬ 
making) 

V 

Time 

requirement 

K 

No  additional  time 
required  to  make 
the  changes/course 
revisions 

Moderate  amount  of  time 

needed  to  make 
changes/course  revisions  (e.g., 
a  few  hours  a  week) 

Extreme  amount  of  time 

needed  to  make 
changes/course  revisions 
(e.g.,  entire  days  per  week 
spent  making  revisions) 

f 

Manpower 

requirement 

K 

Changes  can 
be/were  made  with 
current  personnel 
(e.g.,  1-2  people) 

Some  additional  manpower 
was  needed  to  make  the 
course  changes/revisions  (e.g., 
accomplished  changes  by 
adjusting  personnel  in-house) 

Much  additional  manpower 
was  needed  to  make  the 
changes/  course  revisions 
(e.g.,  had  to  hire  outside 
personnel  with  specific  skills 
to  accomplish  the  changes 
such  as  software  developers) 

V 

Workload 

K 

Workload  of 

individuals  remained 
the  same  during  the 
planning  process 

Workload  of  individuals 
increased  moderately  during 
the  planning  process. 

Workload  of  individuals 
increased  substantially 
during  the  planning 
process. 

—  •> 

V 

Costs 

K 

Inexpensive  to  make 
changes/course 
revisions  (e.g.,  no  new 
costs  to  make 
changes) 

Somewhat  costly  to  make 
changes/course  revisions  (e.g., 
required  some  additional  costs 
to  make  planned 
changes/course  revisions) 

Very  costly  to  make 
changes/course  revisions 
(e.g.,  required  much 
additional  funds  to  make 
the  changes/course 
revisions) 

,> 

V 
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4.  Soldier  Competencies 


All  individuals  answer  "A":  To  the  best  of  your  knowledge,  please  circle  your  response  to  indicate  which 
Soldier  Competencies  were  addressed  in  the  plan  to  change/revise  the  course. 

Training  Developers  also  answer  "B":  Please  circle  your  response  to  indicate  the  Soldier  Competencies 
which  you  integrated  into  the  course. 

Course  Monitors  also  answer  "C":  Please  circle  your  response  to  indicate  the  Soldier  Competencies  that 
you  monitored  being  integrated  into  the  course. 


SOLDIER  COMPETENCIES 

Developers  and 
Monitors 

Developers  Only 

Monitors  Only 

A)  Was  this 
competency  in 
the  plan? 

B)  Did  you  work  on 
integrating  this 
competency  into 
the  course? 

C)  Did  you  monitor 
work  on  integrating 
this  competency  into 
the  course? 

a.  Tactical  and  Technical 
Competence 

Y  N  DNK 

Y  N 

Y  N 

b.  Character  and  Accountability 

Y  N  DNK 

Y  N 

Y  N 

c.  Comprehensive  Fitness 

Y  N  DNK 

Y  N 

Y  N 

d.  Adaptability  and  Initiative 

Y  N  DNK 

Y  N 

Y  N 

e.  Teamwork  and  Collaboration 

Y  N  DNK 

Y  N 

Y  N 

f.  Lifelong  Learner  (includes 
digital  literacy) 

Y  N  DNK 

Y  N 

Y  N 

g.  Communication  and 
Engagements  (oral,  written, 
negotiation) 

Y  N  DNK 

Y  N 

Y  N 

h.  Critical  Thinking  and  Problem 
solving 

Y  N  DNK 

Y  N 

Y  N 

i.  Cultural  and  Joint  Interagency, 
Intergovernmental,  and 
Multinational  competence 

Y  N  DNK 

Y  N 

Y  N 
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5.  Methods  of  Instructional  Delivery 

The  Army  Learning  Model  cites  many  methods  by  which  courses  can  be  conducted. 

All  individuals  answer  "A":  To  the  best  of  your  knowledge,  please  circle  your  response  to  indicate  which 
methods  of  instructional  delivery  were  addressed  in  the  plan  to  change/revise  the  course. 

Training  Developers  also  answer  "B":  Please  circle  your  response  to  indicate  the  delivery  methods  on 
which  you  worked. 

Course  Monitors  also  answer  "C":  Please  circle  your  response  to  indicate  the  instructional  development 
activities  that  you  monitored. 


METHODS  OF  INSTRUCTIONAL 

DELIVERY 

Developers  & 
Monitors 

Developers  Only 

Monitors  Only 

A)  Was  this  area  in 
the  plan? 

B)  Did  you  work  on 
this  area? 

C)  Did  you  monitor 
this  area? 

a.  Instructor  led 

Y  N  DNK 

Y  N 

Y  N 

b.  PowerPoint  slides 

Y  N  DNK 

Y  N 

Y  N 

c.  Hands-on  exercises  with 
equipment 

Y  N  DNK 

Y  N 

Y  N 

d.  Problem-solving  exercises 
(not  technology-based) 

Y  N  DNK 

Y  N 

Y  N 

e.  Training  devices 

Y  N  DNK 

Y  N 

Y  N 

f.  Prerequisite  lessons  prior  to 
course  (distance  learning) 

Y  N  DNK 

Y  N 

Y  N 

Technology-based 

g.  Blended  Learning 

Y  N  DNK 

Y  N 

Y  N 

h.  Simulations/  Gaming 

Y  N  DNK 

Y  N 

Y  N 

i.  Mobile  applications 

Y  N  DNK 

Y  N 

Y  N 

j.  CBT/IMI  lessons 

Y  N  DNK 

Y  N 

Y  N 

k.  Intelligent  tutors 

Y  N  DNK 

Y  N 

Y  N 

Other  ( please  write  in) 

Y  N  DNK 

Y  N 

Y  N 
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6.  Other  ALM  Concepts 


All  individuals  answer  "A":  To  the  best  of  your  knowledge,  please  circle  your  response  to  indicate  which 
areas  were  addressed  in  the  plan  to  change/revise  the  course. 

Training  Developers  also  answer  "B":  please  circle  your  response  to  indicate  the  area  in  which  you 
worked. 

Course  Monitors  also  answer  "C":  please  circle  your  response  to  indicate  the  areas  or  activities  that  you 
monitored. 


OTHER  AREAS  STRESSED  IN  ALM 

Developers  and 
Monitors 

Developers 

Only 

Monitors 

Only 

A)  Was  this  area  in 
the  plan? 

B)  Did  you  work 
in  this  area? 

C)  Did  you  monitor 
work  in  this  area? 

a.  Learner-centered  activities 

Y  N  DNK 

Y  N 

Y  N 

b.  Problem-solving  events  and 
activities  (foster  ability  to  solve 
problems,  to  think,  to  adapt,  to  take 
initiative...) 

Y  N  DNK 

Y  N 

Y  N 

c.  Making  the  course  more  relevant 
to  the  field  (to  Soldier's  future  duty 
assignment) 

Y  N  DNK 

Y  N 

Y  N 

d.  Changing  or  enhancing  feedback 
to  students 

Y  N  DNK 

Y  N 

Y  N 

e.  Techniques/Procedures  to 
improve  the  retention  of  skills  or 
knowledge  of  course  content 

Y  N  DNK 

Y  N 

Y  N 
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Course  Monitors/Managers  only:  Manpower  and  Time  Estimates 


7.  For  each  of  the  major  areas  covered  in  this  interview,  please  circle  your  response  to  indicate 
the  number  of  training  developers  working  in  that  area  either  full  time  or  part-time. 

a.  Enter  a  0  when  no  one  was  working  either  full  time  or  part-time  (e.g.,  if  all  were  full¬ 
time,  then  enter  0  for  part-time).  Leave  cells  blank  when  no  changes  were  made  in 
an  area. 


Manpower  Estimates 

#  of  Training 

Was  the  manpower  more  than 

AREAS  OF  REVISIONS/CHANGES 

Developers 

expected,  same  as  expected  or  less 
than  expected? 

Full-time 

Part-time 

a.  Outcomes  Statements 

MORE 

SAME 

LESS 

b.  Soldier  Competencies 

MORE 

SAME 

LESS 

c.  Course  Content  -  subject  matter 

MORE 

SAME 

LESS 

d.  Tailored  Training 

MORE 

SAME 

LESS 

e.  Methods  of  Instructional  Delivery 

MORE 

SAME 

LESS 

f.  Learner-centered  and/or  Problem¬ 
solving  activities/exercises/scenarios 

MORE 

SAME 

LESS 

g.  Principles  of  Learning  to  enhance 
learning,  retention,  transfer  to  the  field 

MORE 

SAME 

LESS 

h.  Other  (please  write  in) 

MORE 

SAME 

LESS 

i.  Resources  involved  in  compiling  the 
revised  course  -  editing,  checking  for 
internal  consistency,  etc. 

MORE 

SAME 

LESS 
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8.  For  each  of  the  major  areas  covered  in  this  interview,  please  circle  your  response  to  indicate 
the  estimated  time  that  was  spent  on  changes  in  each  area  (consider  start  to  completion 
dates,  e.g.,  from  April  through  May  would  be  2  months).  Leave  cells  blank  when  no  changes 
were  made  in  an  area. 


Time  Estimates 

AREAS  OF  REVISIONS/CHANGES 

Time 

Estimate 

Was  the  time  more  than 
expected,  the  same  as  expected 

or  less  than  expected 

a.  Outcomes  Statements 

MORE 

SAME 

LESS 

b.  Soldier  Competencies 

MORE 

SAME 

LESS 

c.  Tailored  Training 

MORE 

SAME 

LESS 

d.  Methods  of  Instructional  Delivery 

MORE 

SAME 

LESS 

e.  Learner-centered  and/or  problem-solving 
activities/exercises/scenarios 

MORE 

SAME 

LESS 

f.  Principles  of  Learning  to  enhance  learning, 
retention,  transfer  to  the  field 

MORE 

SAME 

LESS 

g.  Other  ( please  write  in) 

MORE 

SAME 

LESS 

h.  Resources  involved  in  compiling  the  revised 
course  -  editing,  checking  for  internal  consistency, 
etc. 

MORE 

SAME 

LESS 
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EXECUTING  THE  REVISED  COURSE  INTERVIEW  QUESTIONS 


Position: _ (filled  in  by  interviewer) 

[Questions  assume  the  revised  course  has  been  executed  at  least  once.] 

You  stated  previously  that  you  were  involved  in  execution/presenting  or  monitoring  of  the  revised 
course. 

The  next  set  of  questions  focuses  on  that  process  and  the  lessons  learned  with  the  revised  course. 
Answer  them  to  the  best  of  your  ability  and  knowledge.  We  understand  that  individuals  had  different 
experiences  and  roles  in  this  process.  If  you  are  unable  to  answer  a  question  based  on  your  experience, 
simply  indicate  that  you  are  not  aware  of  what  happened. 


Interviewer:  There  are  two  sets  of  questions.  Ask  instructors  the  first  set  of  questions. 
Ask  course  monitors/managers  the  second  set  of  questions. 


Instructor  Background  Questions 

1.  Briefly  describe  your  role  in  the  Execution  stage _ 

2.  Were  you  an  instructor  for  this  course  prior  to  the  revisions  being  made?  Yes  No 

If  yes,  how  many  times  did  you  teach  it? _ 

3.  How  many  times  have  you  taught  the  revised  course? 

4.  When  and  how  were  you  told  about  the  approaches  to  be  used  in  the  revised  course? 

5.  How  soon  was  this  prior  to  when  the  revised  course  was  first  executed? 

6.  What  communications  did  you  have  with  the  training  developers  during  course  execution? 
a.  Did  you  have  any  opportunity  to  work  with  training  developers  during  their  redesign? 

7.  Did  you  receive  guidance  throughout  the  course  from  any  outside  personnel?  Or  did  you 
execute  the  course  as  was  planned? 

8.  How  did  you  interact  with  the  course  manager  during  implementation  of  the  course? 

Course  execution 

9.  Did  you  have  a  tryout  or  "dry-run"  of  the  course? 

a.  If  yes,  what  was  the  timeframe  of  the  first  iteration  of  the  course  changes? 
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b.  If  no  dry  run,  did  you  have  sufficient  time  to  prepare  for  the  course  with  the  revised 
changes?  Did  you  have  an  opportunity  to  ask  questions  of  training  developers,  if 
needed? 

10.  Were  the  lesson  plans  such  that  you  clearly  understood  the  intent  of  the  changes? 

11.  If  new  technologies  were  inserted,  were  you  prepared  to  use  them  effectively? 

12.  How  do  you  feel  that  your  instructional  methods  have  changed  in  executing  the  revised 
course? 

a.  Did  your  methods  change  substantially  from  before  the  changes?  Describe  any  changes, 
and  what  impacted  them. 

b.  Did  you  use  other  methods  in  the  execution  of  the  course  that  were  not  originally 
planned  for  or  developed?  Describe  any  changes,  and  what  impacted  them. 

13.  How  do  you  feel  that  your  interactions  with  students  have  changed  in  executing  the  revised 
course?  Did  you  consider  yourself  as  a  "facilitator"  in  any  parts  of  the  revised  course, 
whereas  previously  you  would  have  been  a  "sage  on  the  stage"  (direct  instruction)? 

14.  Did  the  course  include  team-based  (collaborative)  or  peer-to-peer  exercises? 

a.  Can  you  describe  the  nature  of  these  exercises? 

b.  When  was  instructor-involvement  necessary  in  such  exercises? 

15.  Did  you  feel  that  any  sections  of  the  course  which  involved  you  as  a  facilitator  or  which 
were  team-based  (collaborative)  or  peer-to-peer  exercises  required  greater  in-depth  subject 
matter  knowledge  than  a  direct  instruction  approach?  Yes  No  Explain  your  answer. 

16.  Did  the  lesson  plans  and  supporting  materials/technologies  allow  you  to  meet  ALM 
objectives  such  as  the  following?  Interviewer:  The  instructor  should  respond  to  each  of 
the  following  areas.  Probe  their  answers. 

a.  Learner-centered  training 

b.  Tailored  training 

c.  Facilitating  the  development  of  Soldier  competencies 

d.  Improving  Soldier  problem-solving,  thinking  abilities,  and/or  initiative 

e.  Making  the  content  more  relevant  to  the  Soldier's  career  field 

f.  Other 

17.  If  Soldier  competencies  were  integrated  in  the  lesson  plans,  what  challenges  were 
presented  in  trying  to  instill  these  competencies  in  the  students? 

Internal  Course  Assessment 

18.  What  have  been  the  student  reactions  to  the  revised  course? 
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a.  Probe:  Are  student  expectations  different?  Have  they  heard  about  the 
changes/aware  of  the  changes? 

b.  Which  procedures  or  technologies  do  students  particularly  like/dislike,  find 
beneficial? 

19.  Were  the  performance  tests  prior  to  the  revisions  the  same  so  you  could  compare 
performance  after  the  revised  version  of  the  course?  Yes  No 

a.  If  yes,  what  are  the  trends  in  the  results? 

b.  If  no,  do  you  feel  that  students  are  better  performers  after  the  course  revisions  were 
made? 

i.  What  student  behaviors  indicate  better  performance? 

ii.  Do  the  students  seem  to  retain  the  information  better?  What  behaviors  indicate 
higher  levels  of  retention? 

iii.  Do  the  students  seem  more  engaged?  What  student  behaviors  indicate  higher 
levels  of  engagement? 

iv.  Do  the  students  seem  more  motivated?  What  student  behaviors  indicate  higher 
levels  of  motivation? 

20.  If  the  lesson  plans  specified  procedures  for  assessing  competencies,  how  did  these 
techniques  work  for  you? 

21.  Did  you  conduct  AARs?  Can  you  describe  exceptionally  good  or  poor  examples  of  AARs? 

Resources 

22.  Did  you  have  the  necessary  resources  to  implement  the  changes? 

a.  Did  you  feel  that  you  needed  additional  instructors  to  implement  the  course? 
i.  Did  you  ask  for  and  receive  additional  instructors? 

b.  What  logistical  support  was  needed  to  execute  the  course?  Was  this  available?  Did 
you  request  additional  support?  If  so,  please  describe. 

c.  Did  your  workload  increase,  decrease,  remain  the  same? 
i.  Describe  changes  and  why  you  think  these  occurred. 

d.  Do  you  feel  you  were  adequately  prepared  to  teach  the  course  with  the 
changes/revisions? 

i.  How  would  you  design  the  instructor  training  now  that  you  have  taught  the 
revised  course?  What  particular  instructional  points/capabilities  should  be 
stressed  for  a  new  instructor  -  one  who  has  never  taught  the  course  previously? 
What  instructional  skills  are  critical? 
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Course  Execution  Lessons  Learned 


23.  What  were  the  main  lessons  learned  from  executing  the  planned  changes? 

24.  Which  changes/course  revisions  worked  well? 

a.  Which  of  the  revisions  to  the  course  did  you  find  to  be  most  effective? 

b.  Which  changes  were  easiest  for  you  to  implement? 

c.  What  features  did  you  think  should  be  sustained? 

25.  What  were  the  main  challenges  in  executing  the  changes? 

26.  What  would  you  have  done  or  are  doing  differently? 

a.  Were  you  given  the  opportunity  to  further  refine  the  course? 

If  so,  what  further  changes  have  you  been  able  to  make? 

27.  Do  you  have  future  changes  planned?  What  are  they?  Timeline? 

28.  What  do  you  feel  are  the  main  sustainment  issues? 

29.  What  other  comments  do  you  have  on  executing  the  revised  course? 

Course  Monitor/Manager  Questions  on  Execution 

1.  Briefly  describe  your  role  in  the  execution  stage. 

a.  What  were  your  interactions  with  instructors  during  course  execution  (while  they  were 
instructing)? 

2.  When  did  you  monitor  the  revised  course? 

a.  Was  this  the  first  iteration  or  a  later  iteration? 

b.  Did  you  observe  the  entire  course  or  only  some  blocks  of  instruction? 

If  only  some  blocks  of  instruction,  which  blocks  did  you  observe? 

3.  How  many  instructors  did  you  observe? 

4.  Based  on  your  observations,  what  are  your  reactions  to  the  revised  course?  If  you  do  not 
have  sufficient  information  to  answer  a  question,  simply  state  that  is  the  case. 

5.  If  new  technologies  were  inserted,  were  the  instructors  able  to  use  them  effectively? 

a.  Did  the  instructional  technologies  seem  to  accomplish  their  intended  purpose  or  are 
revisions  needed? 

b.  Do  you  believe  that  the  instructional  methods  used  by  the  instructors  changed  when  the 
revised  course  was  executed?  If  so.  How? 

c.  Do  you  believe  that  the  instructors'  interactions  with  students  changed  in  executing  the 
revised  course? 
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d.  If  the  course  included  team-based  or  peer-to-peer  exercises,  please  describe  the  nature 
of  these  exercises. 

e.  For  the  appropriate  lessons,  was  the  instructor  able  to  change  from  being  a  "sage  on  the 
stage"  to  being  a  "facilitator"? 

i.  Was  this  easily  accomplished  or  did  the  instructor  appear  to  need  more 
preparation? 

ii.  Did  these  lessons  appear  to  demand  in-depth  subject  matter  knowledge  on  part 
of  the  instructor? 

f.  How  did  the  lesson  plans  and  supporting  materials/technologies  appear  to  meet  such 
ALM  objectives  as  the  following?  Interviewer:  The  individual  should  respond  to  each 
item.  Probe 

i.  Learner-centered  training 

ii.  Tailored  training 

iii.  Facilitating  the  development  of  Soldier  competencies 

iv.  Improving  Soldier  problem-solving,  thinking  abilities,  and/or  initiative 

v.  Making  the  content  more  relevant  to  the  Soldier's  career  field 

vi.  Other 

6.  If  Soldier  competencies  were  integrated,  do  you  think  the  integration  was  effective? 

7.  How  did  the  ALM  changes  impact  the  logistical  support  for  the  course?  Were  the  logistical 
requirements  of  the  course/the  instructors  met? 

Course  Execution  Lessons  Learned 

8.  From  your  perspective,  what  were  the  main  lessons  learned  from  executing  the  planned 
changes? 

9.  Which  changes/course  revisions  seemed  to  work  well? 

a.  Which  of  the  revisions  to  the  course  did  you  find  to  be  most  effective? 

b.  Which  features  do  you  think  should  be  sustained? 

10.  What  were  the  main  challenges  in  executing  the  changes? 

11.  What  POI  lessons  do  you  think  should  be  revised? 

12.  What  other  changes  would  you  recommend? 

13.  Did  the  instructors  seem  to  be  adequately  prepared  to  teach  the  revised  course? 

14.  How  would  you  design  the  instructor  training  now  that  you  have  observed  course 
execution? 
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Probes.  What  particular  instructional  points/capabilities  should  be  stressed  when  preparing 
a  new  instructor  -  one  who  has  never  taught  the  course  previously?  What  instructional 
skills  are  critical?  In  what  ways  do  these  skills  differ  from  the  current  set  of  skills  which  are 
emphasized? 

15.  What  do  you  feel  are  the  main  sustainment  issues? 

a.  Probes:  Did  you  have  the  necessary  resources  to  implement  the  changes?  What 
additional  resources  are  needed  for  successful  and  continuing  implementation? 

16.  Have  you  or  do  you  plan  to  provide  feedback  on  implementation  to  training  developers, 
course  planners  and/or  key  decision-makers?  If  feedback  has  been  given,  what  were  the 
major  points? 

a.  What  lessons  learned  would  you  like  to  share  regarding  the  process  of  ALM 
implementation? 

External  Assessment 

17.  Have  any  external  assessments  been  conducted  on  student  reactions  to  the  revised  course? 
a.  If  yes,  what  were  the  major  findings?  Did  the  findings  imply  that  further  course 

modifications  are  needed? 

18.  Have  any  external  assessments  been  conducted  on  student  performance/proficiency  in  the 
revised  course?  Is  there  an  appropriate  baseline  measure  that  could  be  used? 

a.  If  so,  who  is  collecting  these  data?  What  are  the  major  findings? 

b.  If  no  comparable  performance  measures  exist,  is  there  a  plan  to  obtain  them  in  the 
future?  What  unit/agency  would  be  responsible  for  these  efforts  -  developing 
measures,  collecting  the  data,  reporting  the  findings,  etc.? 

c.  From  your  observations,  do  you  feel  that  students  are  better  performers  after  the 
course  revisions  were  made?  What  student  behaviors  indicate  better  performance? 

d.  Do  the  students  seem  to  retain  the  information  better?  What  behaviors  indicate  higher 
levels  of  retention? 

e.  Do  the  students  seem  more  engaged?  What  student  behaviors  indicate  higher  levels  of 
engagement? 

f.  Do  the  students  seem  more  motivated?  What  student  behaviors  indicate  higher  levels 
of  motivation? 

General  Reactions 

19.  Has  the  implementation  of  the  ALM  changes  affected  your  work  load?  If  so,  in  what  way? 

20.  What  other  comments  do  you  have  on  executing  the  revised  course? 
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SUSTAINMENT  OF  COURSE  CHANGES 


Position: _ (filled  in  by  interviewer) 

In  answer  to  a  previous  question,  you  indicated  that  you  would  probably  have  a  role  in  sustaining  the 
course  changes.  Once  changes  are  made  to  courses,  it  is  sometimes  hard  to  ensure  than  they  are 
sustained. 

Roles/Responsibilities 

1.  First,  what  will  be  (is)  your  role  in  sustaining  the  course  changes? 

2.  Has  the  course  been  executed  several  times?  Yes  No 

a.  If  yes,  how  many?  We  consider  3x  sustainment. 

3.  What  agencies/units/individuals  will  be/are  involved  with  sustaining  the  change? 

4.  What  will  be/are  their  specific  responsibilities? 

5.  What  agency/unit  will  be/is  responsible  for  the  following?  If  not  known  at  this  time,  please 
indicate  that  this  is  unknown. 

a.  Obtaining  information  on: 

student  performance? _ 

student  reactions? _ 

b.  Obtaining  recommendations  regarding  course  improvements: 

from  instructors? _ 

from  training  developers? _ 

c.  Obtaining  input  on  ways  to  improve  the  course  from  the  instructors? _ 

d.  Determining  whether  instructors  are  executing  the  revised  course  as 

intended? _ 

e.  Ensuring  new  instructors  are  prepared  to  execute  the  revised  course  as 

intended? _ 

f.  Checking  to  determine  whether  content  needs  to  be  updated  and  the  impact  that 

has  on  the  methods  of  delivering  the  training  (e.g.,  lesson  plans,  technology 
update)? _ 

g.  Determining  a  solution  to  a  particular  lesson  or  technological  approach  when  it  is 

not  working  as  desired? _ 

h.  Sustaining  required  training  support/logistics? _ 

6.  If  you  identified  your  agency  as  responsible  for  one  or  more  of  these  actions,  what 
challenges  do  you  envision  with  carrying  out  these  responsibilities?  What  plans  have  been 
developed  to  address  sustainment  issues? 
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7.  Often  with  course  changes,  the  impetus  or  rationale  for  the  changes  "disappear"  after  the 
individuals  originally  involved  in  initiating  the  changes  are  no  longer  present  (that  is,  they 
"disappear"  as  well).  What  plans  will  be/have  been  established  for  a-lasting  audit  trail  of 
the  rationale  for  course  changes,  of  the  critical  decisions  that  were  made,  and  for  updating 
this  audit  trail? 

a.  Is/was  there  a  Knowledge  Management  Officer  or  NCO  assigned  to  the  course? 

If  the  answer  to  #  2  is  Yes  and  3x  or  more,  then  ask  the  following  questions  regarding  Overall  Lessons 
Learned  from  Course  Execution  (Questions  8-12) 

8.  What  was  learned  about  refinements  to  the  revised  course  over  this  period? 

9.  What  was  learned  about  consistency  in  course  execution  over  time? 

10.  What  challenges  existed  regarding  sustaining  course  changes? 

11.  During  this  period  of  time,  what  type  of  additional  support  was  requested  by: 

a.  Faculty/staff 

b.  Training  developers 

c.  Instructors 

d.  Students 

e.  Quality  assurance 

12.  What  other  resources  do  you  estimate  would  be  needed  to  sustain  the  changes? 

a.  Do  you  think  that  additional  personnel  will  be  required  to  sustain  the  changes? 

_ Yes  _ No 

If  yes,  how  many  additional  personnel? _ 

b.  Similarly,  do  you  feel  that  the  current  personnel  have  the  appropriate  skills  to 

sustain  the  changes? _ Yes  _ No 

If  no,  what  additional  skills  are  needed  to  sustain  the  revisions? 

c.  How  do  you  think  sustainment  of  the  changes  will  affect  individual  workloads? 
More,  less,  about  the  same? 
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Appendix  C 

Codebook  for  Interviews 


Code 

Sub-Code 

Shot-in-the-Arm  Initiative 

SITA  Attendance 

SITA  Take  Away 

•  U nclear TRADOC  guidance 

Course  Selection 

Involvement  in  Course  Selection 

Course  Selection 

Front-end  Analysis 

Role 

Role  of  Course  Manager 

Role  of  Middle  Manager 

Role  of  Staff  &  Faculty 

•  Sustainment 

•  Communication 

•  Descriptions  of  roles 

•  Materials/knowledge  for  Staff  and  Faculty 

Role  of  QA 

•  Sustainment 

•  Communication 

•  Descriptions  of  roles 

•  Unclearrole  of  Quality  Assurance  Personnel 

Champion 

•  Any  direct  mention  of  a  person  who  spearheaded  the 
efforts  and  what  position. 

Instructor 

Instructor  Turnover 

•  Personnel  turbulence  of  key  positions 

Instructor  Workload 

Knowledge  of  ALM 

Knowledge 

•  Did  they  have  knowledge  or  have  to  acquire 
knowledge? 

•  Effects  of  having  to  get  knowledge  (workload) 

•  Difficulty  in  acquiring  knowledge  (no  access  to 
classes) 

•  Uncertain  of  changes 

•  Asking  for  further  clarification  of  ALM 

•  Did  they  have  the  necessary  skillsets  to  make  the 
revisions  by  position 

•  Did  they  relate  it  to  something  they  have  done 
previously  or  currently  do  (e.g.,  prior  courses,  prior 
instructional  techniques,  deployed  experiences) 

•  What  ALM  stands  for 

•  Prior  background  knowledge  of  students 

Instructor  Training 

•  Instructor  preparation 
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Training  Developer T raining 

Communication  between  Stakeholders 

Communication 

•  Communication  between  key  positions 

•  Interactions  with  training  developers 

Rate  of  Adop 

tion  Variables 

Compatibility 

Complexity 

•  Workload 

•  Degree  of  difficulty  in  creating  exercises 

•  Degree  of  difficulty  in  tailoring  training 

•  Degree  of  difficulty  in  making  the  course  student 
centered 

•  Degree  of  difficulty  in  peer  coaching 

•  Degree  of  difficulty  in  developing  and  using 
technology 

•  Difficulty  in  making  course  relevant  to  the  field 

•  Difficulty  in  improving  student  performance 

•  Degree  of  change  -  entire  course,  specific  lessons 
within  the  course,  specific  exercises,  homework  only, 
reduction  in  PPT  only,  other  specific  changes  only. 

•  Assessment  challenges 

•  Extent  of  feedback  to  students 

Observability 

•  Student  behaviors 

•  Instructors  behaviors 

•  Course  manager  behaviors 

•  Leader  behaviors 

•  Sustainment  processes 

•  Technology  improvements 

•  Additional  resources  provided 

•  Comparison  to  legacy  course 

•  Changes  in  course  outcomes 

Re-invention 

•  Instructional  techniques 

•  Defining  ALM  for  themselves 

•  Types  of  revisions 

•  Extent  of  revisions 

•  Training  gap  identified 

•  Modify 

•  Rejection  of  some  components  of  ALM 

•  ALM  perceived  as  flexible,  adaptable  conversely 
doesn't  fit,  doesn't  apply 

•  Second  or  third  attempts  of  revisions 

•  Instructor  writers 

•  Making  pen  and  ink  changes 

•  Flexibility  to  make  changes  on  own 

•  Rewrite 

•  Restructure 

•  Change 

•  Revise 

•  Redo 
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Relative  Advantage 

•  Positive  comments  regarding  changes 

•  Negative  comments  regarding  changes 

•  Perceptions  of  sustainment 

•  Comparison  to  pre-SITA 

•  Incentives  to  change 

Trialability 

•  To  what  degree  were  different  things  tried  in  the 
course? 

•  Survey  data  had  a  question  regarding  how  often  the 
revised  course  was  executed. 

•  Pilot? 

Other  Activities  that  Impact  Stage 

•  Teachable  moments 

•  Resource  constraints 

•  Group  testing 

•  Instructors  have  to  understand  the  material 

•  Prior  background  knowledge  of  students 

•  Interactions  with  training  developers 

21st  Century  Soldier  Competencies 

Adaptability  &  Initiative 

•  Initiative 

Character  &  Accountability 

Communication  &  Engagement 

Comprehensive  Fitness 

Critical  Thinking  &  Problem  Solving 

•  Problem  solving 

Cultural  &  J IIM 

Lifelong  Learner 

Tactical  ^Technical 

Teamwork  &  Collaboration 

Types  of  Revisions 

Blended  Learning 

Capstone  Exercises 

Facilitation 

•  Facilitate 

Group  Activities 

•  Break  into  small  groups 

•  Team  based 

Hands-on  Experiences 

•  Hands-on 

Homework 

•  Homework 

•  Work  outside  of  class 

•  Pre-reading 

Learner  centric 

•  Learner  centric 

•  Student  centered 

Peer-to-Peer 

•  Pair  strong  and  weak  students 

PowerPoint 

•  Reduction  in  PowerPoint 

Practical  Exercises 

•  Scenario-based 

•  Experiential  learning 

Pre  &  Post  Testing 

•  Pre-assessment 

•  Pop  quizzes 

Reinforcement 

•  Reinforce  information  and  materials 

Rubric 

Tailored  Training 

•  Tailored  training 

Technology 
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Training  Development  Capacity 

Training  Development  Capability 

•  Training  Development  Capability  or  TDC 

Outcomes 

Major  Accomplishments 

•  Changes  in  student  behaviors  for  the  positive 

•  Positive  comments  by  position 

•  Metthe  intent  of  ALM  tenets 

•  Success  stories  from  bullets 

•  Positive  feedback 

Student  Related  Activities  &  Outcomes 

•  Howto  find  information 

•  How  to  use  technical  manuals 

•  Conduct  research 

•  Student  participation 

•  Keeps  awake 

•  Want  to  learn 

•  Application 

•  Be  prepared 

•  Participate 

•  Fail  a  test 

•  Asking  questions 

•  Don't  like  it 

•  Didn't  study 

•  Quality  of  briefs  goes  down 

•  Information  not  retained 

Student  Feedback 

Receiving  U nit F eedback 
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Appendix  D 

Responses  to  the  Background,  Planning,  and  Developing  Revisions  Questionnaires 

Background  Participant  Questionnaire 


Stage  1.  Decision  Making 

Please  indicate  on  the  scale  below  the  best  description  of  the  decision-making  process. 


#  of  Respondents 

Response  Options 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Candidate  courses  hard  to  identify;  decision  made  after 

much  discussion  and  debate 

0 

0 

0 

Several  courses  immediately  considered;  pros  and  cons 
weighed  before  the  final  decision 

1 

5 

2 

Candidate  course(s)  easily  identified.  Course  decision  made 

with  little  debate 

5 

1 

3 

Stage  2.  Planning  and  Identifying  Solutions 

Please  indicate  the  degree  to  which  the  following  activities  are(have)  occurring(ed)  during  the  process  of 
planning  for  course  revisions. 


Response  Options  for  Identifying  Revisions 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Issue/revisions  not  identified  or  agreed  upon  by  the  team  as 

the  most  critical  to  change 

0 

1 

1 

Issues/revisions  somewhat  identified  and  agreed  upon  by 
the  team  as  the  most  critical  to  change 

0 

5 

5 

Issues/revisions  fully  identified  and  agreed  upon  as  the  most 

critical  to  change 

6 

2 

2 

Response  Options  For  Identifying  Solutions 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Proposed  solutions/changes  did  not  match  the  issues  that 
were  identified;  team  did  not/has  not  agree(d)  on 

solutions/changes 

0 

2 

0 

Proposed  solutions/changes  somewhat  matched  the 
identified  issues;  team  somewhat  agreed  on 

solutions/changes 

0 

3 

6 

Proposed  solutions/changes  matched/were  appropriate  for 
the  identified  issues;  team  fully  agreed  on  solutions/changes 

6 

3 

1 
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Stage  3:  Developing  Course  Revisions 

Please  indicate  the  degree  which  the  following  activities  are  (have)  occuring(ed)  during  the  process  of 
developing  the  course  revisions. 


Response  Options  for  Developing  Course  Materials 

SUSTAIN3 

EXECUTE 

PLAN/DEV 

Have  not  yet  developed  course  materials 

0 

3 

5 

Have  developed  draft  course  materials 

1 

1 

1 

Have  developed  and  finalized  course  materials 

5 

5 

0 

Response  Options  for  Developing  Instructor  Materials 

SUSTAIN3 

EXECUTE 

PLAN/DEV 

Have  not  yet  developed  materials  for  training  instructors 

1 

3 

2 

Have  developed  draft  materials  for  instructor  training 
Have  fully  developed  instructor  training  and  executed  an 

0 

2 

3 

instructor  training  course 

2 

4 

0 

a  Did  not  tally  a  single  2.5  on  SUSTAIN  for  both  questions,  as  appropriate  category  could  not  be 
determined  (drafted  materials  vs.  fully  drafted/execute). 
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Planning  Questionnaire 


How  were  changes  viewed  by  your  team  in  the  planning  process? 


Degree  of  change 


#  of  Respondents 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Minor  course  revisions:  Slight  adjustments  to  the  current 

lesson  plans 

0 

0 

0 

Moderate  course  revisions:  Additions  or  revisions  of  student 
exercises,  slight  changes  to  instructor  training,  etc. 

1 

0 

1 

Large-scale  course  revisions:  Re-write  of  the  POI  and  lesson 

plans,  extensive  instructor  training 

2 

3 

3 

Difficulty  of  making  the  changes/  course  revisions 


SUSTAIN 

EXECUTE 

PLAN/DEV 

Simple  tasks  to  make  changes/  course  revisions  (e.g.,  low 

fidelity,  no  content  change) 

0 

0 

0 

Moderate  difficulty  tasks  to  make  changes/  course  revisions 
(e.g.,  changes  to  current  content,  changes  to  existing 

software) 

2 

1 

2 

Extremely  difficult/very  complex  to  make  changes/  course 
revisions  (e.g,  new  software  development,  all  new 

content/lesson  plans) 

1 

2 

2 

Change  in  Instructor  behavior/  procedures 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Minor  and  few  changes  needed  in  instructional  techniques 
from  the  prior  course;  no  new  instructor  training  (e.g., 
removal  of  Power  Point  slides,  more  repetition  of  exercises 
to  enhance  learning  and  retention) 

0 

0 

0 

Somewhat  difficult  for  the  instructor.  Most  desired  changes 
built  upon  instructional  techniques  previously  used. 
Instructor  training  needed  for  some  blocks  of  instruction 
(e.g.,  desk-top  simulations,  insertion  of  more  complex 

problem-solving  exercises) 

2 

3 

2 

Substantial  changes  in  instructional  techniques;  training 
required  to  prepare  instructors  for  most  of  the  course  (e.g., 
SUSTAIN  direct  instruction  to  primarily  facilitator,  focus  on 
technical  skills  only  to  focusing  on  decision-making) 

1 

0 

2 
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Time  requirement 


SUSTAIN 

EXECUTE 

PLAN/DEV 

No  additional  time  required  to  make  the  changes/  course 

revisions 

0 

0 

0 

Moderate  amount  of  time  needed  to  make  changes/course 

revisions  (e.g.,  a  few  hours  a  week) 

2 

0 

0 

Extreme  amount  of  time  needed  to  make  changes/course 
revisions  (e.g.,  entire  day  per  week  spent  making  revisions) 

1 

3 

4 

Manpower  requiremen 

t 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Changes  can  be/were  made  with  current  personnel  (e.g,  1-2 

people) 

2 

2 

1 

Some  additional  manpower  was  needed  to  make  the  course 
changes/revisions  (e.g.,  accomplished  changes  by  adjusting 

personnel  in-house) 

1 

0 

1 

Much  additional  manpower  was  needed  to  make  the 
changes/course  revisions  (e.g,  had  to  hire  outside  personnel 
with  specific  skills  to  accomplish  the  changes  such  as 

software  developers) 

0 

1 

2 

Workload  requirement 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Workload  of  individuals  remained  the  same  during  the 

planning  process 

0 

0 

1 

Workload  of  individuals  increased  moderately  during  the 

planning  process 

3 

3 

1 

Workload  of  individuals  increased  substantially  during  the 

planning  process 

0 

0 

2 

Costs 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Inexpensive  to  make  changes/course  revisions  (e.g,  no  new 

costs  to  make  changes) 

3 

0 

1 

Somewhat  costly  to  make  changes/  course  revisions  (e.g, 
required  some  additional  costs  to  make  planned 
changes/course  revisions) 

0 

1 

2 

Very  costly  to  make  changes/course  revisions  (e.g.,  required 
much  additional  funds  to  make  the  changes/course 

revisions) 

0 

1 

0 
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Developing  Revisions  Questionnaire 

Question  3:  How  were  changes  viewed  by  your  team  during  development? 

Degree  of  Change 


#  of  Respondents 


Response  Options  to  Degree  of  Change 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Minor  course  revisions:  Slight  adjustments  to  the  current 

lesson  plans 

0 

3 

1 

Moderate  course  revisions:  Additions  or  revisions  of  student 
exercises,  slight  changes  to  instructor  training,  etc. 

4 

2 

3 

Large-scale  course  revisions:  Re-write  of  the  POI  and  lesson 

plans,  extensive  instructor  training 

3 

5 

2 

Difficulty  of  making  the  changes/  course  revisions 


Response  Options  to  Difficulty  of  Making  Changes/  Revisions  SUSTAIN 
Simple  tasks  to  make  changes/  course  revisions  (e.g.,  low 

fidelity,  no  content  change)  1 

Moderate  difficulty  tasks  to  make  changes/  course  revisions 
(e.g.,  changes  to  current  content,  changes  to  existing 

software)  5 

Extremely  difficult/very  complex  to  make  changes/  course 
revisions  (e.g.,  new  software  development,  all  new 

content/lesson  plans)  1 

Change  in  Instructor  behavior/  procedures 

Response  Options  to  Change  in  Instructor  Behavior/ 

Procedures 

Minor  and  few  changes  needed  in  instructional  techniques 
from  the  prior  course;  no  new  instructor  training  (e.g., 
removal  of  Power  Point  slides,  more  repetition  of  exercises 
to  enhance  learning  and  retention) 

Somewhat  difficult  for  the  instructor.  Most  desired  changes 
built  upon  instructional  techniques  previously  used. 

Instructor  training  needed  for  some  blocks  of  instruction 
(e.g.,  desk-top  simulations,  insertion  of  more  complex 

problem-solving  exercises) 

Substantial  changes  in  instructional  techniques;  training 
required  to  prepare  instructors  for  most  of  the  course  (e.g., 

SUSTAIN  direct  instruction  to  primarily  facilitator,  focus  on 
technical  skills  only  to  focusing  on  decision-making) 
a  A  single  1.5  was  not  tallied. 


SUSTAIN  EXECUTE3  PLAN/DEV 


3  2  1 


2  4  4 


2  3  1 


EXECUTE  PLAN/DEV 
2  1 

5  2 

3  3 
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Time  requirement 


Response  Options  to  Time  Requirement 

SUSTAIN 

EXECUTE3 

PLAN/DEV 

No  additional  time  required  to  make  the  changes/  course 

revisions 

0 

2 

0 

Moderate  amount  of  time  needed  to  make  changes/course 

revisions  (e.g.,  a  few  hours  a  week) 

5 

1 

3 

Extreme  amount  of  time  needed  to  make  changes/course 
revisions  (e.g.,  entire  day  per  week  spent  making  revisions) 

2 

6 

3 

a  A  single  2.5  was  not  tallied. 

Manpower  requiremen 

t 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Changes  can  be/were  made  with  current  personnel  (e.g,  1-2 

people) 

2 

3 

4 

Some  additional  manpower  was  needed  to  make  the  course 
changes/revisions  (e.g.,  accomplished  changes  by  adjusting 

personnel  in-house) 

5 

3 

0 

Much  additional  manpower  was  needed  to  make  the 
changes/course  revisions  (e.g,  had  to  hire  outside  personnel 
with  specific  skills  to  accomplish  the  changes  such  as 

software  developers) 

0 

4 

2 

Workload  requirement 

Response  Options  to  Workload  Requirement 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Workload  of  individuals  remained  the  same  during  the 

planning  process 

2 

1 

1 

Workload  of  individuals  increased  moderately  during  the 

planning  process 

5 

5 

1 

Workload  of  individuals  increased  substantially  during  the 

planning  process 

0 

4 

4 
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Costs 


Response  Options  to  Costs 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Inexpensive  to  make  changes/course  revisions  (e.g,  no  new 

costs  to  make  changes) 

7 

2 

4 

Somewhat  costly  to  make  changes/  course  revisions  (e.g, 
required  some  additional  costs  to  make  planned 
changes/course  revisions) 

0 

3 

1 

Very  costly  to  make  changes/course  revisions  (e.g.,  required 
much  additional  funds  to  make  the  changes/course 

revisions) 

0 

4 

0 

Manpower  and  Time  Estimates  (Seven  areas  were  covered  in  the  questionnaire.  The  questions  also 
asked  the  respondent  to  indicate  the  number  of  full  time  and  part-time  training  developers  working  on 
each  type  of  revision.) 

Question  7:  For  each  of  the  major  areas  covered  in  this  interview,  please  circle  your  response  to  indicate 
the  number  of  training  developers  working  in  that  area  either  full  time  or  part-time. 

Outcome  statements 


#  Respondents 


Was  The  Manpower  More  Than  Expected,  Same  As 

Expected  Or  Less  Than  Expected?  SUSTAIN  EXECUTE  PLAN/DEV 


More 

1 

3 

2 

Same 

2 

3 

0 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,1, 1,1 

2,3 

1 

Part  Time 

1,4 

1 

0 

Soldier  competencies 

#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 

EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

2 

1 

Same 

2 

4 

1 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,1, 1,1 

1,2 

1 

Part  Time 

1,4 

2 

0 
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Course  content  -  subject  matter 

#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 

EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

3 

1 

Same 

2 

3 

1 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,14,1 

2, 2, 2, 3 

1 

Part  Time 

1,4 

2 

0 

Tailored  training 

#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 

EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

3 

1 

Same 

2 

2 

1 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1444 

2 

1 

Part  Time 

1,4 

2 

0 

Methods  of  instructional  delivery 

#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 

EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

3 

1 

Same 

2 

3 

1 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1444 

1,2 

1 

Part  Time 

1,4 

1,2 

0 
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Learner  centered  and/or  problem  solving  activities/exercises/scenarios 


#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 
EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

5 

2 

Same 

2 

1 

0 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,1, 1,1 

1,2 

1 

Part  Time 

1,4 

1,2 

0 

Principles  of  learning  to  enhance  learning,  retention,  transfer  to  the  field 


#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 
EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

3 

2 

Same 

2 

2 

0 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,1, 1,1 

2 

1 

Part  Time 

1,4 

1,2 

0 

Resources  involved  in  compiling  the  revised  course  -  editing,  checking  for  internal 

consistency,  etc. 


#  Respondents 

WAS  THE  MANPOWER  MORE  THAN  EXPECTED,  SAME  AS 
EXPECTED  OR  LESS  THAN  EXPECTED? 

SUSTAIN 

EXECUTE 

PLAN/DEV 

More 

1 

3 

2 

Same 

2 

1 

0 

Less 

1 

0 

0 

#  Training  Developers 

SUSTAIN 

EXECUTE 

PLAN/DEV 

Full  Time 

1,1,1 

0 

1 

Part  Time 

1 

0 

0 
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Appendix  E 

Effects  of  Facilitating  and  Inhibiting  Factors 


Philosophy/Approach  to  Making  Course  Changes 

The  approach  or  plan  for  making  changes  impacted  the  process  of  adoption  and 
consequently  the  rate  of  adoption.  The  existence  of  pilots  facilitated  the  rate  of  change  and 
enhanced  the  degree  to  which  training  developers  and  instructors  communicated  with  each  other 
and  were  on  the  “same  sheet  of  music,”  thereby  facilitating  understanding  and  adoption  of  ALM 
concepts.  Instructors  who  were  initially  reluctant  to  change  gradually  were  able  to  implement 
ALM  approaches  such  as  facilitation  over  repeated  iterations  of  the  course.  On  the  other  hand, 
when  no  such  plan  existed  and /  or  the  plan  was  to  wait  to  implement  until  the  entire  course  was 
revised,  feedback  from  instructors  was  delayed,  and  communication  between  training  developers 
and  instructors  was  often  inhibited.  These  factors  led  to  delays  in  implementation. 


Figure  E-l.  Impacts  of  the  philosophy  /  approach  to  making  course  changes. 
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Increased  Student  Accountability 

Increased  student  accountability  was  reflected  in  the  shift  to  a  facilitation  mode  of 
classroom  instruction  and  in  testing  procedures.  In  both  cases,  students  were  held  more 
accountable  for  their  learning,  rather  than  being  highly  dependent  upon  the  instructors.  Read- 
aheads  and  homework  were  stressed.  If  a  student  had  not  done  the  read-aheads  or  homework, 
he/she  was  not  prepared  for  class  discussions  and  could  not  participate.  Similarly,  as  some 
instructors  tested  material  that  was  not  discussed  or  presented  in  class,  doing  the  necessary 
homework  and  study  was  essential  to  doing  well  on  tests.  Thus  there  were  consequences  for  not 
doing  this  work.  On  the  other  hand,  this  required  more  preparation  on  part  of  the  instructors. 
Figure  E-2  presents  these  different  effects. 


Figure  E-2.  Impacts  of  making  students  more  accountable  for  their  learning. 
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Varied  Student  Backgrounds  and  Experiences 

Diverse  student  backgrounds  were  common  in  many  of  the  courses.  When  instructors 
were  able  to  adapt  to  diverse  student  backgrounds  through  peer-to-peer  systems  or  with  special 
attention,  ALM  concepts  were  fostered.  But  in  other  cases,  time-constraints  and  /  or  the  subject 
matter  prevented  tailoring  to  either  the  low-performing  students  or  the  more  capable  students. 
Facilitation  was  also  difficult  under  such  circumstances  (see  Figure  E-3). 


Figure  E-3.  Impacts  of  varied  student  backgrounds  and  experiences. 
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Communication/Collaboration  between  Training  Developers  and  Instructors 

A  common  issue  with  the  ALM  changes  was  getting  the  developers  and  instructors  to 
have  similar  understandings  of  ALM.  This  required  collaboration  between  these  two  key  groups 
of  individuals,  who  were  often  under  different  chains  of  command.  When  collaboration  was 
established,  particularly  early  in  the  process  or  when  the  relationship  between  the  two  groups 
was  strengthened  over  several  iterations  of  the  revised  course,  the  change  process  was  more 
effective  and  faster  (see  Figure  E-4).  Final  recommendations  are  that  this  collaboration  process 
be  established  in  the  very  first  phases  of  major  curriculum  changes. 


Figure  E-4.  Impacts  of  communication  /  collaboration  between  training  developers  and 
instructors. 
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Resources 


The  primary  resource  demand  for  the  ALM  changes  related  to  personnel,  as  no  new 
technologies  were  developed  for  the  courses  in  the  research.  Wherever  there  were  shortages  of 
personnel,  there  was  a  negative  impact  on  the  innovation  process  (see  Figure  E-5). 


+  Low  Need:  Funds 
for  Instructional 
Materials 


Hi 

-  High  Need: 
Additional  Instructors 
&  TDs  Required 


+  Easy  to  Revise  & 
Adapt  to  Lower 
Fidelity  Materials 


+  Increased  ALM 
Implementation 


-  Increased  Instructor 
and  TD  Workload 

■ 


Figure  E-5.  Impacts  of  resources 
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Instructor  Role  Changes 

When  instructors  also  became  writers  (i.e.,  developers)  as  was  the  case  in  a  few  courses, 
both  positive  and  negative  impacts  were  felt.  These  are  summarized  in  Figure  E-6.  Both  the 
positive  and  negative  impact  should  be  considered  when  this  approach  to  curriculum  change  is 
proposed. 


Figure  E-6. 


Impacts  when  instructors  also  become  writers  of  lesson  plans. 
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